2024-01-21T07:51:01.507 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T07:51:01.507 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi --verbose --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259 -- /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259/orig.config.yaml 2024-01-21T07:51:01.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T07:51:01.654 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259 branch: wip-yuri4-testing-2024-01-18-1257-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7524259' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-01-18-1257-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-18-1257-pacific sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8186 sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-01-18-1257-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErmfK9ULo+YAVrDB/iWuZ9mEXrI/tSfGQ79aCe8Jdwcwh8rShrRBJ/taeB2cBaNGhxolrBtlfEAU19Dp3gh3/Y= smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFmBer2yio2wXRcDtxhYMDLIm+dz3HdB2yr808l5E8NhhkCDQSgjH08RqWzHwrlWNoqYvSDUB6+Q18ZwcZPirE= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_23:17:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-21T07:51:01.654 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa; will attempt to use it 2024-01-21T07:51:01.655 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks 2024-01-21T07:51:01.655 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T07:51:01.656 INFO:teuthology.task.internal:Checking packages... 2024-01-21T07:51:01.680 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec' 2024-01-21T07:51:01.680 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T07:51:01.680 INFO:teuthology.packaging:ref: None 2024-01-21T07:51:01.680 INFO:teuthology.packaging:tag: None 2024-01-21T07:51:01.680 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T07:51:01.680 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:51:01.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T07:51:01.835 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.gea310589 2024-01-21T07:51:01.836 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T07:51:01.843 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T07:51:01.843 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T07:51:01.852 INFO:teuthology.task.internal:Saving configuration 2024-01-21T07:51:01.870 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T07:51:01.874 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T07:51:01.898 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:45:56.367258', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErmfK9ULo+YAVrDB/iWuZ9mEXrI/tSfGQ79aCe8Jdwcwh8rShrRBJ/taeB2cBaNGhxolrBtlfEAU19Dp3gh3/Y='} 2024-01-21T07:51:01.924 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:45:56.368679', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFmBer2yio2wXRcDtxhYMDLIm+dz3HdB2yr808l5E8NhhkCDQSgjH08RqWzHwrlWNoqYvSDUB6+Q18ZwcZPirE='} 2024-01-21T07:51:01.924 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T07:51:01.930 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-21T07:51:01.930 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-21T07:51:01.930 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T07:51:02.003 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f59e59f4160>, signals=[15]) 2024-01-21T07:51:02.003 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T07:51:02.010 INFO:teuthology.task.internal:Opening connections... 2024-01-21T07:51:02.011 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-01-21T07:51:02.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:51:02.090 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-01-21T07:51:02.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:51:02.162 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T07:51:02.168 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-01-21T07:51:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-01-21T07:51:02.191 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-01-21T07:51:02.252 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-01-21T07:51:02.252 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T07:51:02.253 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T07:51:02.254 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-01-21T07:51:02.290 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-01-21T07:51:02.313 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-01-21T07:51:02.313 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T07:51:02.375 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T07:51:02.376 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-01-21T07:51:02.399 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T07:51:02.411 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T07:51:02.416 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T07:51:02.416 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-01-21T07:51:02.419 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-01-21T07:51:02.439 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T07:51:02.445 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T07:51:02.445 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-01-21T07:51:02.481 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-01-21T07:51:02.508 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T07:51:02.567 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T07:51:02.581 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T07:51:02.581 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T07:51:02.582 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T07:51:02.582 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T07:51:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:51:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:51:02.604 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-01-21T07:51:02.604 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-01-21T07:51:02.665 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-489.el8.x86_64 2024-01-21T07:51:02.665 INFO:teuthology.task.kernel:Running kernel on smithi046: 4.18.0-489.el8.x86_64 2024-01-21T07:51:02.665 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-01-21T07:51:02.666 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-489.el8.x86_64 2024-01-21T07:51:02.666 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.18.0-489.el8.x86_64 2024-01-21T07:51:02.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-01-21T07:51:06.330 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - AppStream 23 MB/s | 26 MB 00:01 2024-01-21T07:51:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - AppStream 14 MB/s | 26 MB 00:01 2024-01-21T07:51:12.297 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-01-21T07:51:12.809 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - BaseOS 9.9 MB/s | 10 MB 00:01 2024-01-21T07:51:14.447 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - Extras 38 kB/s | 18 kB 00:00 2024-01-21T07:51:14.967 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2024-01-21T07:51:15.140 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-01-21T07:51:15.475 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-01-21T07:51:15.727 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 6.9 kB 00:00 2024-01-21T07:51:15.970 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-01-21T07:51:16.953 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-01-21T07:51:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-01-21T07:51:20.975 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 465 kB/s | 24 kB 00:00 2024-01-21T07:51:21.396 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 427 kB/s | 24 kB 00:00 2024-01-21T07:51:24.073 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T07:51:24.073 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T07:51:24.194 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T07:51:24.194 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:Install 3 Packages 2024-01-21T07:51:24.195 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T07:51:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 89 M 2024-01-21T07:51:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 96 M 2024-01-21T07:51:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T07:51:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T07:51:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T07:51:24.457 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T07:51:24.457 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T07:51:24.457 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T07:51:24.457 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout:Install 3 Packages 2024-01-21T07:51:24.458 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T07:51:24.459 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 89 M 2024-01-21T07:51:24.459 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 96 M 2024-01-21T07:51:24.459 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T07:51:25.260 INFO:teuthology.orchestra.run.smithi046.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-01-21T07:51:25.714 INFO:teuthology.orchestra.run.smithi078.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 9.5 MB/s | 10 MB 00:01 2024-01-21T07:51:26.485 INFO:teuthology.orchestra.run.smithi046.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-01-21T07:51:26.889 INFO:teuthology.orchestra.run.smithi078.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-01-21T07:51:27.002 INFO:teuthology.orchestra.run.smithi046.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2024-01-21T07:51:27.003 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:51:27.003 INFO:teuthology.orchestra.run.smithi046.stdout:Total 32 MB/s | 89 MB 00:02 2024-01-21T07:51:27.423 INFO:teuthology.orchestra.run.smithi078.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2024-01-21T07:51:27.423 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:51:27.423 INFO:teuthology.orchestra.run.smithi078.stdout:Total 30 MB/s | 89 MB 00:02 2024-01-21T07:51:27.713 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T07:51:28.109 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T07:51:28.143 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T07:51:28.143 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T07:51:28.540 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T07:51:28.540 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T07:51:29.451 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T07:51:29.467 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T07:51:29.775 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T07:51:29.796 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T07:51:32.379 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T07:51:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T07:51:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:51:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:51:36.622 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:51:36.623 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:51:37.014 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:51:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:51:40.463 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:51:40.691 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:51:40.735 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:51:40.983 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:02.576 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:02.929 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:02.929 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:52:02.929 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:52:03.451 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:03.451 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T07:52:03.451 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T07:52:03.451 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T07:52:03.451 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T07:52:03.451 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T07:52:03.451 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T07:52:03.612 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:52:03.878 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:52:03.878 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:52:03.878 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T07:52:03.878 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T07:52:03.878 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T07:52:03.878 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-21T07:52:03.878 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T07:52:03.878 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:52:04.087 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:52:04.088 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:52:04.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.02 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 129.el8 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Group : System Environment/Base 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 9265586 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T07:52:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:Relocations : (not relocatable) 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : CentOS BuildSystem 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T07:52:04.122 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T07:52:04.123 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T07:52:04.123 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 4.18.0-535.el8.x86_64 2024-01-21T07:52:04.123 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T07:52:04.312 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:04.337 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:04.705 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:04.705 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:52:04.705 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:52:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:52:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T07:52:05.222 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T07:52:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T07:52:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T07:52:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T07:52:05.222 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T07:52:05.389 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:52:05.596 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-01-21T07:52:05.615 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:52:05.616 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:52:05.616 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T07:52:05.616 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T07:52:05.616 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T07:52:05.617 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-21T07:52:05.617 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T07:52:05.617 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:52:05.836 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:52:05.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:52:05.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.02 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 129.el8 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Group : System Environment/Base 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 9265586 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-01-21T07:52:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Relocations : (not relocatable) 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : CentOS BuildSystem 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T07:52:05.872 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-01-21T07:52:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T07:52:05.873 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T07:52:05.873 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T07:52:05.873 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 4.18.0-535.el8.x86_64 2024-01-21T07:52:05.874 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T07:52:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-01-21T07:52:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-01-21T07:52:07.824 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-01-21T07:52:07.850 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.2fjjpSPUCm 2024-01-21T07:52:07.851 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2fjjpSPUCm 2024-01-21T07:52:07.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.2fjjpSPUCm 2024-01-21T07:52:08.066 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.2fjjpSPUCm is 6KB 2024-01-21T07:52:08.114 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.2fjjpSPUCm 2024-01-21T07:52:08.131 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T07:52:08.203 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T07:52:08.203 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T07:52:08.203 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T07:52:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T07:52:08.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T07:52:08.380 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-01-21T07:52:09.588 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-01-21T07:52:09.591 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-01-21T07:52:09.613 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.rDxqsmqNer 2024-01-21T07:52:09.613 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rDxqsmqNer 2024-01-21T07:52:09.689 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.rDxqsmqNer 2024-01-21T07:52:09.837 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.rDxqsmqNer is 6KB 2024-01-21T07:52:09.887 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.rDxqsmqNer 2024-01-21T07:52:09.905 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T07:52:09.971 INFO:teuthology.orchestra.run.smithi046.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T07:52:09.971 INFO:teuthology.orchestra.run.smithi046.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T07:52:09.972 INFO:teuthology.orchestra.run.smithi046.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T07:52:09.972 INFO:teuthology.orchestra.run.smithi046.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T07:52:09.973 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T07:52:10.139 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-01-21T07:52:38.408 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-21T07:52:38.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-01-21T07:52:38.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:52:40.144 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-21T07:52:40.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-21T07:52:40.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:52:56.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-01-21T07:52:58.688 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-01-21T07:53:05.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-01-21T07:53:05.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:53:07.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-21T07:53:07.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:53:08.960 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-01-21T07:53:10.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-01-21T07:53:20.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-01-21T07:53:20.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:53:22.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-21T07:53:22.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:53:39.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-01-21T07:53:54.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-01-21T07:53:54.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:53:55.484 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-01-21T07:53:56.258 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-01-21T07:53:56.258 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-21T07:53:56.258 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-01-21T07:53:56.275 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-535.el8.x86_64 2024-01-21T07:53:56.275 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T07:53:56.275 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T07:53:56.275 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T07:53:57.276 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-21T07:53:57.276 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T07:53:57.346 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-01-21T07:53:57.359 DEBUG:teuthology.parallel:result is None 2024-01-21T07:54:22.754 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T07:54:37.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-21T07:54:37.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:54:38.064 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-01-21T07:54:38.374 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-21T07:54:38.375 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-21T07:54:38.375 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-01-21T07:54:38.394 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-535.el8.x86_64 2024-01-21T07:54:38.394 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T07:54:38.394 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T07:54:38.394 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T07:54:39.396 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-21T07:54:39.396 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T07:54:39.495 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-01-21T07:54:39.509 DEBUG:teuthology.parallel:result is None 2024-01-21T07:54:39.510 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T07:54:39.521 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T07:54:39.521 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T07:54:39.523 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T07:54:39.545 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T07:54:39.580 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T07:54:39.594 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T07:54:39.594 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T07:54:39.596 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T07:54:39.636 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T07:54:39.644 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T07:54:39.644 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T07:54:39.673 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T07:54:39.706 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:54:39.717 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:54:39.723 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:54:39.735 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:54:39.736 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T07:54:39.743 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T07:54:39.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T07:54:39.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T07:54:39.792 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T07:54:39.802 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T07:54:39.802 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T07:54:39.832 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T07:54:39.852 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:54:39.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:54:39.987 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:54:40.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:54:40.116 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T07:54:40.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T07:54:40.184 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:54:40.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:54:40.292 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:54:40.382 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:54:40.429 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T07:54:40.429 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T07:54:40.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-01-21T07:54:40.505 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-01-21T07:54:40.561 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T07:54:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T07:54:41.005 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T07:54:41.013 INFO:teuthology.task.internal:Starting timer... 2024-01-21T07:54:41.013 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T07:54:41.049 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T07:54:41.060 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-21T07:54:41.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-01-21T07:54:41.155 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-01-21T07:54:41.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-01-21T07:54:41.231 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-01-21T07:54:41.233 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T07:54:41.233 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-01-21T07:54:41.265 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-01-21T07:54:41.265 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-01-21T07:54:41.298 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-01-21T07:54:41.298 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi078.front.sepia.ceph.com': 'permissive'} 2024-01-21T07:54:41.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T07:54:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:54:41.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T07:54:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:54:41.367 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T07:54:41.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-01-21T07:54:41.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-01-21T07:54:41.435 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T07:54:41.444 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-21T07:54:41.450 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-21T07:54:41.575 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T07:54:41.594 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T07:54:41.596 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi078.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T08:00:37.421 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi078.front.sepia.ceph.com')] 2024-01-21T08:00:37.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-21T08:00:37.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T08:00:37.499 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-01-21T08:00:37.571 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-21T08:00:37.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-01-21T08:00:37.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T08:00:37.642 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-01-21T08:00:37.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-01-21T08:00:37.710 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T08:00:37.721 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T08:00:37.721 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T08:00:37.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T08:00:37.724 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T08:00:37.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T08:00:37.758 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T08:00:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T08:00:37.780 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T08:00:37.794 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T08:00:37.834 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-01-21T08:00:37.846 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-01-21T08:00:37.849 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-01-21T08:00:37.860 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-01-21T08:00:37.861 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T08:00:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T08:00:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T08:00:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T08:00:37.950 INFO:teuthology.orchestra.run.smithi046.stderr:bash: ntpq: command not found 2024-01-21T08:00:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2024-01-21T08:00:37.954 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T08:00:37.954 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-01-21T08:00:37.954 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.955 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.955 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.955 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.956 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T08:00:37.956 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-01-21T08:00:37.956 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.956 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.957 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.957 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:00:37.957 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T08:00:37.966 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T08:00:37.967 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi046.front.sepia.ceph.com 2024-01-21T08:00:37.967 DEBUG:teuthology.orchestra.run.smithi046:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T08:00:37.967 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi078.front.sepia.ceph.com 2024-01-21T08:00:37.967 DEBUG:teuthology.orchestra.run.smithi078:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T08:00:38.749 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:43 ago on Sun 21 Jan 2024 07:58:55 AM UTC. 2024-01-21T08:00:38.785 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:36 ago on Sun 21 Jan 2024 07:59:02 AM UTC. 2024-01-21T08:00:40.677 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:00:40.678 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:00:40.678 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:00:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:00:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:Resetting modules: 2024-01-21T08:00:40.679 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools 2024-01-21T08:00:40.679 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:00:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:00:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:00:40.679 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:00:40.870 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout:Resetting modules: 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout: container-tools 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:00:40.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:00:40.923 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:00:41.142 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:00:41.760 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:46 ago on Sun 21 Jan 2024 07:58:55 AM UTC. 2024-01-21T08:00:41.987 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:39 ago on Sun 21 Jan 2024 07:59:02 AM UTC. 2024-01-21T08:00:43.672 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:00:43.707 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:00:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-01-21T08:00:43.707 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:00:43.707 INFO:teuthology.orchestra.run.smithi078.stdout:Installing group/module packages: 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T08:00:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:Installing module profiles: 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools/common 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools rhel8 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:Install 13 Packages 2024-01-21T08:00:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:00:43.711 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 27 M 2024-01-21T08:00:43.711 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 89 M 2024-01-21T08:00:43.711 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:00:44.054 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:00:44.092 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:00:44.092 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-21T08:00:44.092 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:00:44.092 INFO:teuthology.orchestra.run.smithi046.stdout:Installing group/module packages: 2024-01-21T08:00:44.092 INFO:teuthology.orchestra.run.smithi046.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T08:00:44.092 INFO:teuthology.orchestra.run.smithi046.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T08:00:44.093 INFO:teuthology.orchestra.run.smithi046.stdout:Installing module profiles: 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout: container-tools/common 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling module streams: 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout: container-tools rhel8 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout:Install 13 Packages 2024-01-21T08:00:44.094 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:00:44.095 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 27 M 2024-01-21T08:00:44.095 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 89 M 2024-01-21T08:00:44.095 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:00:44.192 INFO:teuthology.orchestra.run.smithi078.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.6 MB/s | 750 kB 00:00 2024-01-21T08:00:44.234 INFO:teuthology.orchestra.run.smithi078.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.1 MB/s | 1.0 MB 00:00 2024-01-21T08:00:44.259 INFO:teuthology.orchestra.run.smithi078.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 565 kB/s | 38 kB 00:00 2024-01-21T08:00:44.309 INFO:teuthology.orchestra.run.smithi078.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.1 MB/s | 241 kB 00:00 2024-01-21T08:00:44.377 INFO:teuthology.orchestra.run.smithi078.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.6 MB/s | 178 kB 00:00 2024-01-21T08:00:44.493 INFO:teuthology.orchestra.run.smithi078.stdout:(6/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 16 MB/s | 9.6 MB 00:00 2024-01-21T08:00:44.518 INFO:teuthology.orchestra.run.smithi078.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 175 kB/s | 25 kB 00:00 2024-01-21T08:00:44.577 INFO:teuthology.orchestra.run.smithi078.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 12 MB/s | 3.9 MB 00:00 2024-01-21T08:00:44.601 INFO:teuthology.orchestra.run.smithi046.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.4 MB/s | 750 kB 00:00 2024-01-21T08:00:44.602 INFO:teuthology.orchestra.run.smithi078.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 863 kB/s | 94 kB 00:00 2024-01-21T08:00:44.635 INFO:teuthology.orchestra.run.smithi046.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.9 MB/s | 1.0 MB 00:00 2024-01-21T08:00:44.734 INFO:teuthology.orchestra.run.smithi046.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 286 kB/s | 38 kB 00:00 2024-01-21T08:00:44.736 INFO:teuthology.orchestra.run.smithi078.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 16 MB/s | 2.5 MB 00:00 2024-01-21T08:00:44.760 INFO:teuthology.orchestra.run.smithi046.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.9 MB/s | 241 kB 00:00 2024-01-21T08:00:44.761 INFO:teuthology.orchestra.run.smithi078.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 331 kB/s | 53 kB 00:00 2024-01-21T08:00:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 700 kB/s | 41 kB 00:00 2024-01-21T08:00:44.953 INFO:teuthology.orchestra.run.smithi078.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 19 MB/s | 8.1 MB 00:00 2024-01-21T08:00:44.954 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:00:44.954 INFO:teuthology.orchestra.run.smithi078.stdout:Total 21 MB/s | 27 MB 00:01 2024-01-21T08:00:44.977 INFO:teuthology.orchestra.run.smithi046.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 821 kB/s | 178 kB 00:00 2024-01-21T08:00:45.103 INFO:teuthology.orchestra.run.smithi046.stdout:(6/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2024-01-21T08:00:45.128 INFO:teuthology.orchestra.run.smithi046.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 165 kB/s | 25 kB 00:00 2024-01-21T08:00:45.186 INFO:teuthology.orchestra.run.smithi046.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 8.7 MB/s | 3.9 MB 00:00 2024-01-21T08:00:45.203 INFO:teuthology.orchestra.run.smithi046.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 936 kB/s | 94 kB 00:00 2024-01-21T08:00:45.218 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:00:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:00:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:00:45.270 INFO:teuthology.orchestra.run.smithi046.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 790 kB/s | 53 kB 00:00 2024-01-21T08:00:45.329 INFO:teuthology.orchestra.run.smithi046.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 18 MB/s | 2.5 MB 00:00 2024-01-21T08:00:45.346 INFO:teuthology.orchestra.run.smithi046.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 543 kB/s | 41 kB 00:00 2024-01-21T08:00:45.480 INFO:teuthology.orchestra.run.smithi046.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 23 MB/s | 8.1 MB 00:00 2024-01-21T08:00:45.480 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:00:45.480 INFO:teuthology.orchestra.run.smithi046.stdout:Total 19 MB/s | 27 MB 00:01 2024-01-21T08:00:45.499 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:00:45.503 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:00:45.754 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:00:45.777 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:00:45.777 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:00:46.041 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:00:46.045 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:00:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:00:46.433 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T08:00:46.612 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T08:00:46.757 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:00:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T08:00:46.968 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T08:00:47.038 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T08:00:47.145 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T08:00:47.615 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T08:00:47.629 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T08:00:47.870 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T08:00:48.463 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T08:00:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T08:00:48.829 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T08:00:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T08:00:49.326 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T08:00:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T08:00:49.707 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T08:00:49.765 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T08:00:49.978 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T08:00:50.225 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T08:00:50.586 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T08:00:50.627 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T08:00:50.791 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T08:00:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T08:00:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T08:00:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T08:00:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T08:00:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T08:00:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T08:00:51.566 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T08:00:51.777 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T08:00:51.944 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T08:00:51.967 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T08:00:51.967 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:00:51.967 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:00:51.967 INFO:teuthology.orchestra.run.smithi078.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:51.967 INFO:teuthology.orchestra.run.smithi078.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:51.967 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:00:51.968 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:00:52.411 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T08:00:52.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T08:00:52.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T08:00:52.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T08:00:52.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T08:00:52.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T08:00:52.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T08:00:52.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T08:00:52.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T08:00:52.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T08:00:52.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T08:00:52.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T08:00:52.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T08:00:52.633 DEBUG:teuthology.parallel:result is None 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:52.922 INFO:teuthology.orchestra.run.smithi046.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T08:00:52.923 INFO:teuthology.orchestra.run.smithi046.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T08:00:52.923 INFO:teuthology.orchestra.run.smithi046.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T08:00:52.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:00:52.923 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:00:53.290 DEBUG:teuthology.parallel:result is None 2024-01-21T08:00:53.290 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T08:00:53.299 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T08:00:53.299 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi046.front.sepia.ceph.com 2024-01-21T08:00:53.299 DEBUG:teuthology.orchestra.run.smithi046:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T08:00:53.300 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi078.front.sepia.ceph.com 2024-01-21T08:00:53.300 DEBUG:teuthology.orchestra.run.smithi078:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T08:00:53.446 DEBUG:teuthology.parallel:result is None 2024-01-21T08:00:53.495 DEBUG:teuthology.parallel:result is None 2024-01-21T08:00:53.495 INFO:teuthology.run_tasks:Running task install... 2024-01-21T08:00:53.503 DEBUG:teuthology.task.install:project ceph 2024-01-21T08:00:53.503 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'}} 2024-01-21T08:00:53.503 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T08:00:53.504 INFO:teuthology.task.install:Using flavor: default 2024-01-21T08:00:53.510 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-21T08:00:53.510 INFO:teuthology.task.install:extra packages: [] 2024-01-21T08:00:53.511 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T08:00:53.511 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T08:00:53.511 INFO:teuthology.packaging:ref: None 2024-01-21T08:00:53.511 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T08:00:53.511 INFO:teuthology.packaging:branch: None 2024-01-21T08:00:53.511 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T08:00:53.711 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T08:00:53.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T08:00:53.714 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T08:00:53.714 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T08:00:53.714 INFO:teuthology.packaging:ref: None 2024-01-21T08:00:53.714 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T08:00:53.714 INFO:teuthology.packaging:branch: None 2024-01-21T08:00:53.714 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T08:00:53.714 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T08:00:53.866 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T08:00:53.866 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T08:00:53.911 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T08:00:53.911 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:00:53.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T08:00:53.945 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T08:00:53.945 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T08:00:53.945 INFO:teuthology.packaging:ref: None 2024-01-21T08:00:53.945 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T08:00:53.945 INFO:teuthology.packaging:branch: None 2024-01-21T08:00:53.945 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T08:00:53.945 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T08:00:53.946 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T08:00:53.947 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T08:00:53.969 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T08:00:53.970 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:00:53.970 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T08:00:53.999 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T08:00:53.999 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T08:00:53.999 INFO:teuthology.packaging:ref: None 2024-01-21T08:00:53.999 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T08:00:53.999 INFO:teuthology.packaging:branch: None 2024-01-21T08:00:53.999 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T08:00:53.999 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T08:00:54.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T08:00:54.068 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T08:00:54.098 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T08:00:54.146 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T08:00:54.165 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-01-21T08:00:54.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-01-21T08:00:54.213 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-01-21T08:00:54.215 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-01-21T08:00:54.430 INFO:teuthology.orchestra.run.smithi078.stdout:76 files removed 2024-01-21T08:00:54.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw 2024-01-21T08:00:54.495 INFO:teuthology.orchestra.run.smithi046.stdout:76 files removed 2024-01-21T08:00:54.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw 2024-01-21T08:00:54.936 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 1.1 MB/s | 74 kB 00:00 2024-01-21T08:00:55.028 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 951 kB/s | 74 kB 00:00 2024-01-21T08:00:55.045 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 214 kB/s | 15 kB 00:00 2024-01-21T08:00:55.129 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 250 kB/s | 15 kB 00:00 2024-01-21T08:00:55.146 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-01-21T08:00:55.213 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 33 kB/s | 1.7 kB 00:00 2024-01-21T08:00:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - AppStream 21 MB/s | 26 MB 00:01 2024-01-21T08:00:58.116 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - AppStream 9.2 MB/s | 26 MB 00:02 2024-01-21T08:01:02.173 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-01-21T08:01:04.331 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-01-21T08:01:04.357 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - BaseOS 8.5 MB/s | 10 MB 00:01 2024-01-21T08:01:05.090 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 6.9 kB 00:00 2024-01-21T08:01:05.867 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - PowerTools 6.9 MB/s | 4.7 MB 00:00 2024-01-21T08:01:06.482 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-01-21T08:01:06.935 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-21T08:01:06.999 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2024-01-21T08:01:08.269 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-01-21T08:01:08.309 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - PowerTools 3.9 MB/s | 4.7 MB 00:01 2024-01-21T08:01:09.402 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2024-01-21T08:01:10.095 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 27 MB/s | 16 MB 00:00 2024-01-21T08:01:12.036 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 439 kB/s | 24 kB 00:00 2024-01-21T08:01:14.170 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 393 kB/s | 24 kB 00:00 2024-01-21T08:01:15.488 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:01:15.491 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:01:15.491 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-01-21T08:01:15.491 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:01:15.491 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:01:15.491 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T08:01:15.492 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T08:01:15.493 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout:Install 25 Packages 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 2 Packages 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 59 M 2024-01-21T08:01:15.494 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:01:16.162 INFO:teuthology.orchestra.run.smithi078.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2024-01-21T08:01:16.187 INFO:teuthology.orchestra.run.smithi078.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-01-21T08:01:16.221 INFO:teuthology.orchestra.run.smithi078.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-21T08:01:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T08:01:16.354 INFO:teuthology.orchestra.run.smithi078.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-01-21T08:01:16.371 INFO:teuthology.orchestra.run.smithi078.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-01-21T08:01:16.397 INFO:teuthology.orchestra.run.smithi078.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T08:01:16.422 INFO:teuthology.orchestra.run.smithi078.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-01-21T08:01:16.447 INFO:teuthology.orchestra.run.smithi078.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-01-21T08:01:16.505 INFO:teuthology.orchestra.run.smithi078.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 368 kB 00:00 2024-01-21T08:01:16.540 INFO:teuthology.orchestra.run.smithi078.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 114 kB 00:00 2024-01-21T08:01:16.682 INFO:teuthology.orchestra.run.smithi078.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-01-21T08:01:16.966 INFO:teuthology.orchestra.run.smithi078.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-01-21T08:01:16.991 INFO:teuthology.orchestra.run.smithi078.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 322 kB/s | 145 kB 00:00 2024-01-21T08:01:17.050 INFO:teuthology.orchestra.run.smithi078.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 751 kB/s | 44 kB 00:00 2024-01-21T08:01:17.083 INFO:teuthology.orchestra.run.smithi078.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-01-21T08:01:17.183 INFO:teuthology.orchestra.run.smithi078.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-01-21T08:01:17.377 INFO:teuthology.orchestra.run.smithi078.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 129 kB/s | 42 kB 00:00 2024-01-21T08:01:17.417 INFO:teuthology.orchestra.run.smithi078.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 141 kB/s | 47 kB 00:00 2024-01-21T08:01:17.467 INFO:teuthology.orchestra.run.smithi078.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 1.3 MB/s | 116 kB 00:00 2024-01-21T08:01:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:(21/27): ndctl-libs-71.1-7.el8.x86_64.rpm 259 kB/s | 80 kB 00:00 2024-01-21T08:01:17.518 INFO:teuthology.orchestra.run.smithi078.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 3.0 MB/s | 306 kB 00:00 2024-01-21T08:01:17.551 INFO:teuthology.orchestra.run.smithi078.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-01-21T08:01:17.577 INFO:teuthology.orchestra.run.smithi078.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 1.6 MB/s | 181 kB 00:00 2024-01-21T08:01:17.602 INFO:teuthology.orchestra.run.smithi078.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 895 kB/s | 75 kB 00:00 2024-01-21T08:01:17.677 INFO:teuthology.orchestra.run.smithi078.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-01-21T08:01:17.752 INFO:teuthology.orchestra.run.smithi078.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2024-01-21T08:01:17.755 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:01:17.755 INFO:teuthology.orchestra.run.smithi078.stdout:Total 26 MB/s | 59 MB 00:02 2024-01-21T08:01:17.826 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:01:17.853 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:01:17.856 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T08:01:17.857 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T08:01:17.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout:Install 25 Packages 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 2 Packages 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:01:17.859 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 59 M 2024-01-21T08:01:17.860 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:01:17.874 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:01:17.874 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:01:18.313 INFO:teuthology.orchestra.run.smithi046.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 38 MB/s | 6.0 MB 00:00 2024-01-21T08:01:18.329 INFO:teuthology.orchestra.run.smithi046.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-01-21T08:01:18.363 INFO:teuthology.orchestra.run.smithi046.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-21T08:01:18.397 INFO:teuthology.orchestra.run.smithi046.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T08:01:18.427 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:01:18.430 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:01:18.497 INFO:teuthology.orchestra.run.smithi046.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-01-21T08:01:18.514 INFO:teuthology.orchestra.run.smithi046.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-01-21T08:01:18.531 INFO:teuthology.orchestra.run.smithi046.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2024-01-21T08:01:18.565 INFO:teuthology.orchestra.run.smithi046.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 213 kB 00:00 2024-01-21T08:01:18.598 INFO:teuthology.orchestra.run.smithi046.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T08:01:18.632 INFO:teuthology.orchestra.run.smithi046.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-21T08:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T08:01:18.824 INFO:teuthology.orchestra.run.smithi046.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-01-21T08:01:19.141 INFO:teuthology.orchestra.run.smithi046.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-01-21T08:01:19.306 INFO:teuthology.orchestra.run.smithi046.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 224 kB/s | 145 kB 00:00 2024-01-21T08:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:01:19.709 INFO:teuthology.orchestra.run.smithi046.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 109 kB/s | 44 kB 00:00 2024-01-21T08:01:19.859 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T08:01:19.859 INFO:teuthology.orchestra.run.smithi046.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 654 kB/s | 677 kB 00:01 2024-01-21T08:01:19.893 INFO:teuthology.orchestra.run.smithi046.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 345 kB/s | 259 kB 00:00 2024-01-21T08:01:20.094 INFO:teuthology.orchestra.run.smithi046.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 110 kB/s | 42 kB 00:00 2024-01-21T08:01:20.096 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T08:01:20.160 INFO:teuthology.orchestra.run.smithi046.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 157 kB/s | 47 kB 00:00 2024-01-21T08:01:20.194 INFO:teuthology.orchestra.run.smithi046.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 267 kB/s | 80 kB 00:00 2024-01-21T08:01:20.266 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T08:01:20.561 INFO:teuthology.orchestra.run.smithi046.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 249 kB/s | 116 kB 00:00 2024-01-21T08:01:20.628 INFO:teuthology.orchestra.run.smithi046.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 417 kB/s | 181 kB 00:00 2024-01-21T08:01:20.670 INFO:teuthology.orchestra.run.smithi046.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 542 kB/s | 59 kB 00:00 2024-01-21T08:01:20.695 INFO:teuthology.orchestra.run.smithi046.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 573 kB/s | 306 kB 00:00 2024-01-21T08:01:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T08:01:20.737 INFO:teuthology.orchestra.run.smithi046.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 689 kB/s | 75 kB 00:00 2024-01-21T08:01:20.821 INFO:teuthology.orchestra.run.smithi046.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2024-01-21T08:01:20.879 INFO:teuthology.orchestra.run.smithi046.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 22 MB/s | 4.0 MB 00:00 2024-01-21T08:01:20.881 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:01:20.881 INFO:teuthology.orchestra.run.smithi046.stdout:Total 20 MB/s | 59 MB 00:03 2024-01-21T08:01:20.924 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T08:01:20.954 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:01:21.008 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:01:21.008 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:01:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T08:01:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T08:01:21.385 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:01:21.388 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:01:21.424 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T08:01:21.490 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T08:01:21.742 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T08:01:22.292 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T08:01:22.466 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T08:01:22.482 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:01:22.546 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T08:01:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T08:01:22.802 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T08:01:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T08:01:22.979 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T08:01:23.009 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T08:01:23.138 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T08:01:23.198 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T08:01:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T08:01:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T08:01:23.581 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T08:01:23.618 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T08:01:23.974 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T08:01:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T08:01:24.138 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T08:01:24.319 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T08:01:24.473 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T08:01:24.542 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T08:01:24.787 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T08:01:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T08:01:25.016 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T08:01:25.121 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T08:01:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T08:01:25.315 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T08:01:25.526 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T08:01:25.657 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T08:01:25.731 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T08:01:25.923 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T08:01:25.958 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T08:01:26.009 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T08:01:26.140 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T08:01:26.204 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T08:01:26.382 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T08:01:26.415 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T08:01:26.473 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T08:01:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T08:01:26.639 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T08:01:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T08:01:26.973 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T08:01:27.051 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T08:01:27.235 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T08:01:27.696 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T08:01:27.856 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T08:01:27.954 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T08:01:28.144 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T08:01:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T08:01:28.498 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T08:01:28.567 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T08:01:28.743 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T08:01:28.957 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T08:01:29.124 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T08:01:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T08:01:29.348 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T08:01:29.382 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T08:01:30.078 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T08:01:30.355 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T08:01:30.473 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T08:01:30.582 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T08:01:32.113 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T08:01:32.365 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T08:01:33.111 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T08:01:33.417 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T08:01:33.532 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T08:01:33.672 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /run 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2024-01-21T08:01:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:01:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T08:01:51.218 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T08:01:51.218 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T08:01:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T08:01:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T08:01:52.155 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T08:01:52.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T08:01:52.157 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T08:01:52.158 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T08:01:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T08:01:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:01:53.104 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-01-21T08:01:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.104 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:01:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T08:01:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:01:53.106 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:01:53.409 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-test 2024-01-21T08:01:53.706 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T08:01:53.706 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /dev 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /run 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /dev 2024-01-21T08:01:53.707 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:01:54.138 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:43 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:01:55.122 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T08:01:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T08:01:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T08:01:55.564 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T08:01:55.564 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T08:01:55.619 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T08:01:55.621 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 53 M 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 229 M 2024-01-21T08:01:55.622 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:01:55.811 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.2 MB/s | 169 kB 00:00 2024-01-21T08:01:55.922 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 823 kB/s | 203 kB 00:00 2024-01-21T08:01:56.253 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T08:01:56.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T08:01:56.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T08:01:56.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T08:01:56.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T08:01:56.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T08:01:56.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T08:01:56.254 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T08:01:56.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T08:01:56.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T08:01:56.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T08:01:56.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T08:01:56.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T08:01:56.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T08:01:56.473 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2024-01-21T08:01:56.498 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 274 kB/s | 188 kB 00:00 2024-01-21T08:01:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 538 kB/s | 323 kB 00:00 2024-01-21T08:01:56.590 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 770 kB/s | 71 kB 00:00 2024-01-21T08:01:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:01:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:Total 54 MB/s | 53 MB 00:00 2024-01-21T08:01:56.635 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:01:56.656 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:01:56.656 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:01:56.974 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:01:56.976 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:01:57.119 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T08:01:57.119 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T08:01:57.120 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:01:57.121 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:01:57.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-test 2024-01-21T08:01:57.454 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:01:57.714 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T08:01:57.901 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T08:01:58.076 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T08:01:58.161 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T08:01:58.183 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:44 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:01:58.352 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T08:01:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T08:01:59.773 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T08:01:59.774 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:01:59.775 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 53 M 2024-01-21T08:01:59.776 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 229 M 2024-01-21T08:01:59.776 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:02:00.029 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2024-01-21T08:02:00.204 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 868 kB/s | 203 kB 00:00 2024-01-21T08:02:00.230 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 940 kB/s | 188 kB 00:00 2024-01-21T08:02:00.280 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T08:02:00.305 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.1 MB/s | 323 kB 00:00 2024-01-21T08:02:00.822 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 61 MB/s | 52 MB 00:00 2024-01-21T08:02:00.824 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:00.824 INFO:teuthology.orchestra.run.smithi046.stdout:Total 50 MB/s | 53 MB 00:01 2024-01-21T08:02:00.868 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:02:00.891 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:02:00.891 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:02:01.223 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:02:01.226 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:02:01.715 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:02:01.941 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T08:02:02.121 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T08:02:02.277 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T08:02:02.361 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T08:02:02.528 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T08:02:02.668 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T08:02:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T08:02:03.618 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T08:02:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T08:02:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T08:02:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T08:02:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T08:02:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T08:02:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T08:02:04.323 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T08:02:04.323 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:04.324 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:02:04.324 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T08:02:04.324 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T08:02:04.324 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T08:02:04.324 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:04.324 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:04.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph 2024-01-21T08:02:05.270 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:54 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:06.749 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:06.752 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-01-21T08:02:06.752 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:06.752 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:02:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T08:02:06.752 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T08:02:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T08:02:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout:Install 32 Packages 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:06.755 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 31 M 2024-01-21T08:02:06.756 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 111 M 2024-01-21T08:02:06.756 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:02:07.234 INFO:teuthology.orchestra.run.smithi078.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 147 kB/s | 6.3 kB 00:00 2024-01-21T08:02:07.309 INFO:teuthology.orchestra.run.smithi078.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-01-21T08:02:07.368 INFO:teuthology.orchestra.run.smithi078.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-01-21T08:02:07.393 INFO:teuthology.orchestra.run.smithi078.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-21T08:02:07.486 INFO:teuthology.orchestra.run.smithi078.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-01-21T08:02:07.752 INFO:teuthology.orchestra.run.smithi078.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 41 MB/s | 18 MB 00:00 2024-01-21T08:02:07.777 INFO:teuthology.orchestra.run.smithi078.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 133 kB/s | 39 kB 00:00 2024-01-21T08:02:07.803 INFO:teuthology.orchestra.run.smithi078.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 384 kB/s | 157 kB 00:00 2024-01-21T08:02:07.823 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T08:02:07.861 INFO:teuthology.orchestra.run.smithi078.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 681 kB/s | 57 kB 00:00 2024-01-21T08:02:07.920 INFO:teuthology.orchestra.run.smithi078.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-01-21T08:02:07.946 INFO:teuthology.orchestra.run.smithi078.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 536 kB/s | 103 kB 00:00 2024-01-21T08:02:08.059 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T08:02:08.313 INFO:teuthology.orchestra.run.smithi078.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 551 kB/s | 249 kB 00:00 2024-01-21T08:02:08.338 INFO:teuthology.orchestra.run.smithi078.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 278 kB/s | 109 kB 00:00 2024-01-21T08:02:08.385 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T08:02:08.385 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T08:02:08.385 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T08:02:08.385 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T08:02:08.385 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T08:02:08.385 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T08:02:08.438 INFO:teuthology.orchestra.run.smithi078.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.2 MB/s | 641 kB 00:00 2024-01-21T08:02:08.663 INFO:teuthology.orchestra.run.smithi078.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 124 kB/s | 44 kB 00:00 2024-01-21T08:02:08.689 INFO:teuthology.orchestra.run.smithi078.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 690 kB/s | 173 kB 00:00 2024-01-21T08:02:08.714 INFO:teuthology.orchestra.run.smithi078.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 493 kB/s | 185 kB 00:00 2024-01-21T08:02:08.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T08:02:08.823 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:08.823 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:02:08.823 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T08:02:08.824 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T08:02:08.824 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T08:02:08.824 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:08.824 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:08.881 INFO:teuthology.orchestra.run.smithi078.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 58 kB/s | 11 kB 00:00 2024-01-21T08:02:08.931 INFO:teuthology.orchestra.run.smithi078.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 81 kB/s | 18 kB 00:00 2024-01-21T08:02:08.998 INFO:teuthology.orchestra.run.smithi078.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 417 kB/s | 49 kB 00:00 2024-01-21T08:02:09.023 INFO:teuthology.orchestra.run.smithi078.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-01-21T08:02:09.057 INFO:teuthology.orchestra.run.smithi078.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 468 kB/s | 59 kB 00:00 2024-01-21T08:02:09.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph 2024-01-21T08:02:09.107 INFO:teuthology.orchestra.run.smithi078.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2024-01-21T08:02:09.166 INFO:teuthology.orchestra.run.smithi078.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2024-01-21T08:02:09.191 INFO:teuthology.orchestra.run.smithi078.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 286 kB/s | 24 kB 00:00 2024-01-21T08:02:09.216 INFO:teuthology.orchestra.run.smithi078.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-01-21T08:02:09.242 INFO:teuthology.orchestra.run.smithi078.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-01-21T08:02:09.284 INFO:teuthology.orchestra.run.smithi078.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2024-01-21T08:02:09.309 INFO:teuthology.orchestra.run.smithi078.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-01-21T08:02:09.418 INFO:teuthology.orchestra.run.smithi078.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2024-01-21T08:02:09.443 INFO:teuthology.orchestra.run.smithi078.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 172 kB/s | 23 kB 00:00 2024-01-21T08:02:09.468 INFO:teuthology.orchestra.run.smithi078.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 464 kB/s | 85 kB 00:00 2024-01-21T08:02:09.469 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:09.469 INFO:teuthology.orchestra.run.smithi078.stdout:Total 11 MB/s | 31 MB 00:02 2024-01-21T08:02:09.549 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:02:09.588 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:02:09.588 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:02:09.826 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:55 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:02:09.904 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:02:09.908 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:02:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:02:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T08:02:11.430 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:02:11.433 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:11.433 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T08:02:11.434 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T08:02:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout:Install 32 Packages 2024-01-21T08:02:11.436 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:11.437 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 31 M 2024-01-21T08:02:11.437 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 111 M 2024-01-21T08:02:11.437 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:02:11.526 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T08:02:11.784 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T08:02:11.915 INFO:teuthology.orchestra.run.smithi046.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 111 kB/s | 6.3 kB 00:00 2024-01-21T08:02:11.965 INFO:teuthology.orchestra.run.smithi046.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-21T08:02:12.024 INFO:teuthology.orchestra.run.smithi046.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-01-21T08:02:12.025 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T08:02:12.049 INFO:teuthology.orchestra.run.smithi046.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-21T08:02:12.133 INFO:teuthology.orchestra.run.smithi046.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-01-21T08:02:12.205 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T08:02:12.375 INFO:teuthology.orchestra.run.smithi046.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-21T08:02:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 150 kB/s | 39 kB 00:00 2024-01-21T08:02:12.417 INFO:teuthology.orchestra.run.smithi046.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 427 kB/s | 157 kB 00:00 2024-01-21T08:02:12.418 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T08:02:12.467 INFO:teuthology.orchestra.run.smithi046.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 758 kB/s | 57 kB 00:00 2024-01-21T08:02:12.534 INFO:teuthology.orchestra.run.smithi046.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-01-21T08:02:12.551 INFO:teuthology.orchestra.run.smithi046.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 588 kB/s | 103 kB 00:00 2024-01-21T08:02:12.604 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T08:02:12.815 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T08:02:12.919 INFO:teuthology.orchestra.run.smithi046.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 297 kB/s | 109 kB 00:00 2024-01-21T08:02:12.944 INFO:teuthology.orchestra.run.smithi046.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 523 kB/s | 249 kB 00:00 2024-01-21T08:02:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T08:02:13.111 INFO:teuthology.orchestra.run.smithi046.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.1 MB/s | 641 kB 00:00 2024-01-21T08:02:13.136 INFO:teuthology.orchestra.run.smithi046.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 201 kB/s | 44 kB 00:00 2024-01-21T08:02:13.169 INFO:teuthology.orchestra.run.smithi046.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 822 kB/s | 185 kB 00:00 2024-01-21T08:02:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T08:02:13.312 INFO:teuthology.orchestra.run.smithi046.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 861 kB/s | 173 kB 00:00 2024-01-21T08:02:13.337 INFO:teuthology.orchestra.run.smithi046.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 66 kB/s | 11 kB 00:00 2024-01-21T08:02:13.404 INFO:teuthology.orchestra.run.smithi046.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 731 kB/s | 49 kB 00:00 2024-01-21T08:02:13.429 INFO:teuthology.orchestra.run.smithi046.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2024-01-21T08:02:13.462 INFO:teuthology.orchestra.run.smithi046.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-01-21T08:02:13.488 INFO:teuthology.orchestra.run.smithi046.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-01-21T08:02:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T08:02:13.521 INFO:teuthology.orchestra.run.smithi046.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-01-21T08:02:13.546 INFO:teuthology.orchestra.run.smithi046.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2024-01-21T08:02:13.572 INFO:teuthology.orchestra.run.smithi046.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2024-01-21T08:02:13.597 INFO:teuthology.orchestra.run.smithi046.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 315 kB/s | 24 kB 00:00 2024-01-21T08:02:13.623 INFO:teuthology.orchestra.run.smithi046.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 386 kB/s | 29 kB 00:00 2024-01-21T08:02:13.648 INFO:teuthology.orchestra.run.smithi046.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-01-21T08:02:13.674 INFO:teuthology.orchestra.run.smithi046.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-01-21T08:02:13.707 INFO:teuthology.orchestra.run.smithi046.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2024-01-21T08:02:13.732 INFO:teuthology.orchestra.run.smithi046.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2024-01-21T08:02:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T08:02:13.757 INFO:teuthology.orchestra.run.smithi046.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 275 kB/s | 23 kB 00:00 2024-01-21T08:02:13.758 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:13.759 INFO:teuthology.orchestra.run.smithi046.stdout:Total 13 MB/s | 31 MB 00:02 2024-01-21T08:02:13.844 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:02:13.885 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:02:13.885 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:02:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T08:02:14.197 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:02:14.200 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:02:14.370 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T08:02:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T08:02:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T08:02:15.052 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T08:02:15.322 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:02:15.337 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T08:02:15.518 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T08:02:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T08:02:15.717 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T08:02:15.779 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T08:02:15.965 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T08:02:15.998 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T08:02:16.174 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T08:02:16.287 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T08:02:16.355 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T08:02:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T08:02:16.689 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T08:02:16.763 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T08:02:16.846 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T08:02:17.024 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T08:02:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T08:02:17.236 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T08:02:17.456 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T08:02:17.463 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T08:02:17.685 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T08:02:17.927 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T08:02:17.969 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T08:02:18.173 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T08:02:18.342 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T08:02:18.579 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T08:02:18.759 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T08:02:19.048 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T08:02:19.362 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T08:02:19.655 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T08:02:19.701 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T08:02:19.875 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T08:02:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T08:02:20.055 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T08:02:20.262 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T08:02:20.347 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T08:02:20.518 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T08:02:20.581 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T08:02:20.755 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T08:02:20.803 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T08:02:20.970 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T08:02:21.017 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T08:02:21.035 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T08:02:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T08:02:21.430 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T08:02:21.591 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T08:02:21.591 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T08:02:21.591 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T08:02:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T08:02:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T08:02:21.594 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T08:02:21.594 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T08:02:21.594 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T08:02:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T08:02:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T08:02:22.411 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T08:02:22.416 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T08:02:22.416 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:22.416 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:02:22.416 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:22.416 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T08:02:22.417 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T08:02:22.418 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:22.419 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:22.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-base 2024-01-21T08:02:23.397 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:12 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:24.037 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T08:02:24.227 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T08:02:24.688 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T08:02:24.826 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:02:24.872 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:24.874 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:02:24.874 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:24.918 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T08:02:25.153 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T08:02:25.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install cephadm 2024-01-21T08:02:25.333 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T08:02:25.352 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T08:02:25.475 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T08:02:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T08:02:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T08:02:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T08:02:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T08:02:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T08:02:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T08:02:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T08:02:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T08:02:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T08:02:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T08:02:25.901 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T08:02:25.901 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T08:02:25.901 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T08:02:25.901 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T08:02:25.901 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T08:02:25.901 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T08:02:25.938 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:14 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:26.801 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T08:02:26.802 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T08:02:26.803 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:26.804 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:27.066 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-base 2024-01-21T08:02:27.425 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:27.426 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:02:27.426 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 71 k 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 290 k 2024-01-21T08:02:27.427 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:02:27.474 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-01-21T08:02:27.474 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:27.475 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.4 MB/s | 71 kB 00:00 2024-01-21T08:02:27.475 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:02:27.482 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:02:27.482 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:02:27.483 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:02:27.487 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:02:27.528 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:02:27.826 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:13 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:02:28.044 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:28.243 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:28.874 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:28.875 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:02:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T08:02:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:28.875 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:29.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T08:02:29.376 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:02:29.425 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:02:29.427 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:02:29.427 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:29.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install cephadm 2024-01-21T08:02:29.851 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:18 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:30.314 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:02:31.308 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:31.309 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-01-21T08:02:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:31.310 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 163 k 2024-01-21T08:02:31.310 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 587 k 2024-01-21T08:02:31.310 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:02:31.713 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 405 kB/s | 163 kB 00:00 2024-01-21T08:02:31.713 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:31.713 INFO:teuthology.orchestra.run.smithi078.stdout:Total 404 kB/s | 163 kB 00:00 2024-01-21T08:02:31.714 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:02:31.725 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:02:31.725 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:02:31.775 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:02:31.779 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:02:31.894 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:02:31.909 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:02:31.910 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 71 k 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 290 k 2024-01-21T08:02:31.911 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:02:31.961 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T08:02:31.962 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:31.962 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.4 MB/s | 71 kB 00:00 2024-01-21T08:02:31.962 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:02:31.970 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:02:31.970 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:02:31.972 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:02:31.975 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:02:32.020 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:02:32.090 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:02:32.502 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:02:32.626 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:32.798 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:02:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:32.898 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:02:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:32.898 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:33.035 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:33.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr 2024-01-21T08:02:33.458 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T08:02:33.459 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:33.459 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:02:33.459 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T08:02:33.459 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:33.459 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:33.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T08:02:33.890 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:34.430 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:20 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:02:35.314 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:02:35.359 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:35.360 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:02:35.361 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:35.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T08:02:36.024 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:02:36.024 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:36.024 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-01-21T08:02:36.024 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:36.025 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 163 k 2024-01-21T08:02:36.026 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 587 k 2024-01-21T08:02:36.026 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:02:36.077 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.1 MB/s | 163 kB 00:00 2024-01-21T08:02:36.077 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:36.078 INFO:teuthology.orchestra.run.smithi046.stdout:Total 3.0 MB/s | 163 kB 00:00 2024-01-21T08:02:36.078 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:02:36.090 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:02:36.090 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:02:36.140 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:02:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:02:36.179 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:25 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:36.255 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:02:36.465 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:02:36.903 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:02:37.265 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:02:37.265 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:37.265 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:02:37.265 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T08:02:37.265 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:37.265 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:37.477 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr 2024-01-21T08:02:37.664 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:37.665 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-01-21T08:02:37.665 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:37.665 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:02:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T08:02:37.665 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2024-01-21T08:02:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:37.667 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 3.0 M 2024-01-21T08:02:37.667 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 15 M 2024-01-21T08:02:37.667 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:02:37.950 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 482 kB/s | 24 kB 00:00 2024-01-21T08:02:37.975 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 119 kB/s | 8.9 kB 00:00 2024-01-21T08:02:38.033 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-01-21T08:02:38.168 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 175 kB/s | 34 kB 00:00 2024-01-21T08:02:38.238 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:02:38.284 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 127 kB/s | 43 kB 00:00 2024-01-21T08:02:38.310 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 713 kB/s | 196 kB 00:00 2024-01-21T08:02:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:Total 4.6 MB/s | 3.0 MB 00:00 2024-01-21T08:02:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:02:38.332 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:02:38.332 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:02:38.495 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:02:38.498 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:02:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:02:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T08:02:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T08:02:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T08:02:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T08:02:39.792 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:02:39.841 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:02:39.843 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:02:39.843 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:39.955 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T08:02:40.671 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T08:02:40.714 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:26 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:02:40.945 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:02:41.252 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:02:41.252 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T08:02:41.253 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T08:02:41.253 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T08:02:41.253 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T08:02:41.253 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T08:02:41.623 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:41.624 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:41.751 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T08:02:42.298 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:02:42.299 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:42.299 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-21T08:02:42.299 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:42.299 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:02:42.299 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T08:02:42.300 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:02:42.300 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T08:02:42.300 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T08:02:42.300 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T08:02:42.300 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T08:02:42.300 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T08:02:42.301 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:42.301 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:02:42.301 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:02:42.301 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-01-21T08:02:42.301 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:42.302 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 3.0 M 2024-01-21T08:02:42.302 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 15 M 2024-01-21T08:02:42.303 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:02:42.481 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:31 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:42.580 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 499 kB/s | 24 kB 00:00 2024-01-21T08:02:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 137 kB/s | 8.9 kB 00:00 2024-01-21T08:02:42.673 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2024-01-21T08:02:42.898 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 134 kB/s | 43 kB 00:00 2024-01-21T08:02:43.065 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 72 kB/s | 34 kB 00:00 2024-01-21T08:02:43.207 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 368 kB/s | 196 kB 00:00 2024-01-21T08:02:43.209 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:43.209 INFO:teuthology.orchestra.run.smithi046.stdout:Total 3.2 MB/s | 3.0 MB 00:00 2024-01-21T08:02:43.221 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:02:43.232 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:02:43.233 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:02:43.408 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:02:43.411 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:02:43.690 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:02:43.905 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T08:02:43.971 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:43.972 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-01-21T08:02:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-01-21T08:02:43.972 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-01-21T08:02:43.972 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:02:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T08:02:43.972 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout:Install 8 Packages 2024-01-21T08:02:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:43.974 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 37 M 2024-01-21T08:02:43.974 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 210 M 2024-01-21T08:02:43.974 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:02:44.158 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T08:02:44.349 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T08:02:44.479 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T08:02:44.497 INFO:teuthology.orchestra.run.smithi078.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 59 MB/s | 8.6 MB 00:00 2024-01-21T08:02:44.748 INFO:teuthology.orchestra.run.smithi078.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-01-21T08:02:44.840 INFO:teuthology.orchestra.run.smithi078.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.9 MB/s | 4.9 MB 00:00 2024-01-21T08:02:44.865 INFO:teuthology.orchestra.run.smithi078.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2024-01-21T08:02:45.041 INFO:teuthology.orchestra.run.smithi078.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.9 MB/s | 3.7 MB 00:00 2024-01-21T08:02:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 34 MB/s | 14 MB 00:00 2024-01-21T08:02:45.424 INFO:teuthology.orchestra.run.smithi078.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2024-01-21T08:02:45.522 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T08:02:45.558 INFO:teuthology.orchestra.run.smithi078.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 333 kB/s | 172 kB 00:00 2024-01-21T08:02:45.558 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:45.558 INFO:teuthology.orchestra.run.smithi078.stdout:Total 23 MB/s | 37 MB 00:01 2024-01-21T08:02:45.790 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:02:45.808 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:02:45.808 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:02:45.832 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:02:46.127 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:02:46.127 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T08:02:46.127 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T08:02:46.127 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T08:02:46.127 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T08:02:46.127 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T08:02:46.131 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:02:46.134 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:02:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T08:02:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:46.545 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:02:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T08:02:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T08:02:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T08:02:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T08:02:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T08:02:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T08:02:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:46.546 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:46.670 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T08:02:46.723 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:02:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T08:02:47.033 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T08:02:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T08:02:47.430 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:33 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:02:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T08:02:48.246 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T08:02:48.859 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T08:02:49.031 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T08:02:49.033 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout:Install 8 Packages 2024-01-21T08:02:49.034 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:49.035 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 37 M 2024-01-21T08:02:49.035 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 210 M 2024-01-21T08:02:49.035 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:02:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T08:02:49.562 INFO:teuthology.orchestra.run.smithi046.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 57 MB/s | 8.6 MB 00:00 2024-01-21T08:02:49.745 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T08:02:49.921 INFO:teuthology.orchestra.run.smithi046.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2024-01-21T08:02:49.979 INFO:teuthology.orchestra.run.smithi046.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.8 MB/s | 225 kB 00:00 2024-01-21T08:02:50.099 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T08:02:50.329 INFO:teuthology.orchestra.run.smithi046.stdout:(4/8): python3-scipy-1.0.0-21.module_el8+762+77 41 MB/s | 14 MB 00:00 2024-01-21T08:02:50.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T08:02:50.471 INFO:teuthology.orchestra.run.smithi046.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.9 MB 00:01 2024-01-21T08:02:50.530 INFO:teuthology.orchestra.run.smithi046.stdout:(6/8): openblas-0.3.15-4.el8.x86_64.rpm 4.2 MB/s | 4.7 MB 00:01 2024-01-21T08:02:50.839 INFO:teuthology.orchestra.run.smithi046.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 470 kB/s | 172 kB 00:00 2024-01-21T08:02:50.914 INFO:teuthology.orchestra.run.smithi046.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2024-01-21T08:02:50.915 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:50.915 INFO:teuthology.orchestra.run.smithi046.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-21T08:02:51.156 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:02:51.176 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:02:51.176 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:02:51.508 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:02:51.511 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:02:52.104 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T08:02:52.156 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:02:52.321 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T08:02:52.462 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T08:02:52.604 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T08:02:53.232 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T08:02:53.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T08:02:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T08:02:53.724 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T08:02:54.037 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T08:02:54.039 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T08:02:54.039 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T08:02:54.039 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T08:02:54.040 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T08:02:54.040 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T08:02:54.040 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T08:02:54.040 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T08:02:54.040 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T08:02:54.171 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T08:02:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T08:02:54.447 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T08:02:54.447 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T08:02:54.447 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T08:02:54.447 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:54.447 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:02:54.556 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-rook 2024-01-21T08:02:54.909 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T08:02:55.237 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T08:02:55.295 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:44 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:02:55.462 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T08:02:56.775 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:02:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:56.776 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T08:02:56.777 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout:Install 13 Packages 2024-01-21T08:02:56.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:02:56.779 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 1.9 M 2024-01-21T08:02:56.779 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 24 M 2024-01-21T08:02:56.779 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:02:56.982 INFO:teuthology.orchestra.run.smithi078.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 333 kB/s | 27 kB 00:00 2024-01-21T08:02:57.084 INFO:teuthology.orchestra.run.smithi078.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 109 kB/s | 20 kB 00:00 2024-01-21T08:02:57.109 INFO:teuthology.orchestra.run.smithi078.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 128 kB/s | 27 kB 00:00 2024-01-21T08:02:57.160 INFO:teuthology.orchestra.run.smithi078.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 712 kB/s | 126 kB 00:00 2024-01-21T08:02:57.202 INFO:teuthology.orchestra.run.smithi078.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 940 kB/s | 110 kB 00:00 2024-01-21T08:02:57.327 INFO:teuthology.orchestra.run.smithi078.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 264 kB/s | 33 kB 00:00 2024-01-21T08:02:57.349 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T08:02:57.401 INFO:teuthology.orchestra.run.smithi078.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 221 kB/s | 16 kB 00:00 2024-01-21T08:02:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 142 kB/s | 43 kB 00:00 2024-01-21T08:02:57.486 INFO:teuthology.orchestra.run.smithi078.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 954 kB/s | 81 kB 00:00 2024-01-21T08:02:57.512 INFO:teuthology.orchestra.run.smithi078.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 385 kB/s | 155 kB 00:00 2024-01-21T08:02:57.545 INFO:teuthology.orchestra.run.smithi078.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2024-01-21T08:02:57.646 INFO:teuthology.orchestra.run.smithi078.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 458 kB/s | 61 kB 00:00 2024-01-21T08:02:57.713 INFO:teuthology.orchestra.run.smithi078.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.6 MB/s | 1.2 MB 00:00 2024-01-21T08:02:57.713 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:02:57.713 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2024-01-21T08:02:57.766 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:02:57.780 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:02:57.780 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:02:57.866 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:02:57.869 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:02:58.396 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:02:58.697 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T08:02:58.723 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T08:02:58.915 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T08:02:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T08:02:59.197 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T08:02:59.243 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T08:02:59.243 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T08:02:59.243 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T08:02:59.243 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T08:02:59.243 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T08:02:59.244 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T08:02:59.244 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T08:02:59.244 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T08:02:59.410 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T08:02:59.654 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T08:02:59.654 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:59.654 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:02:59.654 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T08:02:59.654 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T08:02:59.654 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T08:02:59.655 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T08:02:59.655 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T08:02:59.655 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T08:02:59.655 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T08:02:59.655 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T08:02:59.655 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:02:59.655 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:02:59.656 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T08:02:59.765 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-rook 2024-01-21T08:02:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T08:03:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T08:03:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T08:03:00.531 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:46 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T08:03:01.826 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T08:03:02.069 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T08:03:02.113 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:02.114 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T08:03:02.115 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout:Install 13 Packages 2024-01-21T08:03:02.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:02.117 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 1.9 M 2024-01-21T08:03:02.117 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 24 M 2024-01-21T08:03:02.117 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:02.269 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T08:03:02.428 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T08:03:02.541 INFO:teuthology.orchestra.run.smithi046.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 746 kB/s | 27 kB 00:00 2024-01-21T08:03:02.624 INFO:teuthology.orchestra.run.smithi046.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 167 kB/s | 20 kB 00:00 2024-01-21T08:03:02.641 INFO:teuthology.orchestra.run.smithi046.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 195 kB/s | 27 kB 00:00 2024-01-21T08:03:02.725 INFO:teuthology.orchestra.run.smithi046.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 687 kB/s | 126 kB 00:00 2024-01-21T08:03:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T08:03:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T08:03:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T08:03:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T08:03:02.750 INFO:teuthology.orchestra.run.smithi046.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 875 kB/s | 110 kB 00:00 2024-01-21T08:03:02.851 INFO:teuthology.orchestra.run.smithi046.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 330 kB/s | 33 kB 00:00 2024-01-21T08:03:02.910 INFO:teuthology.orchestra.run.smithi046.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 280 kB/s | 16 kB 00:00 2024-01-21T08:03:03.001 INFO:teuthology.orchestra.run.smithi046.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 888 kB/s | 81 kB 00:00 2024-01-21T08:03:03.026 INFO:teuthology.orchestra.run.smithi046.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 142 kB/s | 43 kB 00:00 2024-01-21T08:03:03.071 INFO:teuthology.orchestra.run.smithi046.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 361 kB/s | 155 kB 00:00 2024-01-21T08:03:03.185 INFO:teuthology.orchestra.run.smithi046.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 405 kB/s | 64 kB 00:00 2024-01-21T08:03:03.211 INFO:teuthology.orchestra.run.smithi046.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 438 kB/s | 61 kB 00:00 2024-01-21T08:03:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T08:03:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:03.223 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T08:03:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T08:03:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:03.224 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:03.244 INFO:teuthology.orchestra.run.smithi046.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.8 MB/s | 1.2 MB 00:00 2024-01-21T08:03:03.245 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:03.245 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-01-21T08:03:03.304 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:03.321 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:03.321 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:03.363 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T08:03:03.416 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:03.419 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:03.930 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:04.090 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:53 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:04.166 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T08:03:04.358 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T08:03:04.574 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T08:03:04.810 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T08:03:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T08:03:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T08:03:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T08:03:05.573 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:05.574 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-01-21T08:03:05.574 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:05.574 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:03:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2024-01-21T08:03:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:05.576 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 5.8 M 2024-01-21T08:03:05.576 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 24 M 2024-01-21T08:03:05.576 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:03:05.729 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2024-01-21T08:03:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T08:03:05.946 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.0 MB/s | 538 kB 00:00 2024-01-21T08:03:06.105 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2024-01-21T08:03:06.663 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 23 kB/s | 21 kB 00:00 2024-01-21T08:03:06.876 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T08:03:07.122 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 48 kB/s | 49 kB 00:01 2024-01-21T08:03:07.444 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T08:03:07.589 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 216 kB/s | 355 kB 00:01 2024-01-21T08:03:07.592 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:07.592 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.9 MB/s | 5.8 MB 00:02 2024-01-21T08:03:07.645 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:03:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:03:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:03:07.753 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:03:07.756 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:03:07.778 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T08:03:07.961 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T08:03:08.051 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:03:08.136 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T08:03:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T08:03:08.455 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T08:03:08.455 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T08:03:08.455 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T08:03:08.455 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T08:03:08.456 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T08:03:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T08:03:08.922 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T08:03:08.923 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T08:03:08.924 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T08:03:08.924 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:08.924 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:09.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T08:03:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T08:03:09.837 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:55 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T08:03:10.190 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T08:03:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:03:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:03:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T08:03:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T08:03:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T08:03:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T08:03:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:11.064 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:11.211 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-fuse 2024-01-21T08:03:11.424 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T08:03:11.425 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:11.426 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 5.8 M 2024-01-21T08:03:11.427 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 24 M 2024-01-21T08:03:11.427 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:11.835 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2024-01-21T08:03:11.932 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:00 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:11.977 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 149 kB/s | 21 kB 00:00 2024-01-21T08:03:12.069 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2024-01-21T08:03:12.128 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.3 MB/s | 355 kB 00:00 2024-01-21T08:03:12.203 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2024-01-21T08:03:12.229 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 307 kB/s | 49 kB 00:00 2024-01-21T08:03:12.229 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:12.229 INFO:teuthology.orchestra.run.smithi046.stdout:Total 7.2 MB/s | 5.8 MB 00:00 2024-01-21T08:03:12.280 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:12.291 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:12.291 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:12.402 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:12.405 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T08:03:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T08:03:13.424 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:13.424 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:13.424 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout:Install 2 Packages 2024-01-21T08:03:13.425 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:13.426 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 968 k 2024-01-21T08:03:13.426 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 3.2 M 2024-01-21T08:03:13.426 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:03:13.656 INFO:teuthology.orchestra.run.smithi078.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 19 MB/s | 885 kB 00:00 2024-01-21T08:03:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 222 kB/s | 83 kB 00:00 2024-01-21T08:03:13.985 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:13.985 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.7 MB/s | 968 kB 00:00 2024-01-21T08:03:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:03:14.016 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:03:14.017 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:03:14.103 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:03:14.106 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:03:14.231 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T08:03:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:03:14.564 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T08:03:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T08:03:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T08:03:14.807 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T08:03:15.001 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:03:15.156 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T08:03:15.156 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T08:03:15.310 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T08:03:15.310 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T08:03:15.310 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T08:03:15.310 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T08:03:15.310 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T08:03:15.310 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T08:03:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T08:03:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:15.544 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T08:03:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:15.545 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:15.726 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T08:03:15.726 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:15.727 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:15.727 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T08:03:15.727 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T08:03:15.727 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T08:03:15.727 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:15.727 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:15.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados-devel 2024-01-21T08:03:15.860 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-fuse 2024-01-21T08:03:16.505 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:05 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:17.987 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:17.987 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:17.987 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-01-21T08:03:17.988 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:17.989 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 131 k 2024-01-21T08:03:17.989 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 439 k 2024-01-21T08:03:17.989 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:03:18.038 INFO:teuthology.orchestra.run.smithi078.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 131 kB 00:00 2024-01-21T08:03:18.038 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:18.039 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.5 MB/s | 131 kB 00:00 2024-01-21T08:03:18.039 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:03:18.044 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:03:18.044 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:03:18.093 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:03:18.097 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:03:18.206 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:03:18.219 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:18.220 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:18.220 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:18.220 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:18.220 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:18.220 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T08:03:18.220 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-21T08:03:18.220 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T08:03:18.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:18.221 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:18.221 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:18.221 INFO:teuthology.orchestra.run.smithi046.stdout:Install 2 Packages 2024-01-21T08:03:18.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:18.221 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 968 k 2024-01-21T08:03:18.221 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 3.2 M 2024-01-21T08:03:18.222 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:18.473 INFO:teuthology.orchestra.run.smithi046.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-01-21T08:03:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:18.707 INFO:teuthology.orchestra.run.smithi046.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 282 kB/s | 83 kB 00:00 2024-01-21T08:03:18.707 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:18.708 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.9 MB/s | 968 kB 00:00 2024-01-21T08:03:18.718 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:18.740 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:18.740 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:18.835 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:18.839 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:19.022 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:19.256 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T08:03:19.435 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T08:03:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:19.842 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T08:03:19.843 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T08:03:20.178 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs2 2024-01-21T08:03:20.269 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T08:03:20.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:20.270 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:20.270 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T08:03:20.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:20.270 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:20.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados-devel 2024-01-21T08:03:20.913 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:09 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:21.241 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:07 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:22.358 INFO:teuthology.orchestra.run.smithi078.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:22.403 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:22.404 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:03:22.405 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:22.509 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs-devel 2024-01-21T08:03:22.822 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:22.823 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:22.823 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 131 k 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 439 k 2024-01-21T08:03:22.824 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:22.877 INFO:teuthology.orchestra.run.smithi046.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-01-21T08:03:22.878 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:22.878 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-01-21T08:03:22.878 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:22.884 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:22.884 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:22.937 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:22.940 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:23.250 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:12 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:23.353 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:23.692 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:23.997 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:24.379 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:24.379 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:24.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs2 2024-01-21T08:03:24.743 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:24.744 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:24.744 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:24.744 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:24.744 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:03:24.744 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T08:03:24.745 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:24.745 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:03:24.745 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:24.745 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-01-21T08:03:24.745 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 24 k 2024-01-21T08:03:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 94 k 2024-01-21T08:03:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:03:24.816 INFO:teuthology.orchestra.run.smithi078.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 342 kB/s | 24 kB 00:00 2024-01-21T08:03:24.817 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:24.817 INFO:teuthology.orchestra.run.smithi078.stdout:Total 336 kB/s | 24 kB 00:00 2024-01-21T08:03:24.817 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:03:24.820 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:03:24.820 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:03:24.823 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:03:24.826 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:03:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:03:25.277 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:25.340 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:11 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:26.510 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:26.510 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:26.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados2 2024-01-21T08:03:26.890 INFO:teuthology.orchestra.run.smithi046.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:26.939 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:03:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:27.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs-devel 2024-01-21T08:03:27.387 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:16 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:27.798 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:13 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:28.861 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:28.906 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:28.907 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:03:28.907 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:29.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librbd1 2024-01-21T08:03:29.388 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 24 k 2024-01-21T08:03:29.389 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 94 k 2024-01-21T08:03:29.390 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:29.739 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:18 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:30.007 INFO:teuthology.orchestra.run.smithi046.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 39 kB/s | 24 kB 00:00 2024-01-21T08:03:30.007 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:30.008 INFO:teuthology.orchestra.run.smithi046.stdout:Total 39 kB/s | 24 kB 00:00 2024-01-21T08:03:30.008 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:30.012 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:30.012 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:30.016 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:30.019 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:30.077 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:30.265 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:30.554 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:30.915 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:30.915 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:30.915 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:30.915 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:30.915 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:30.915 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:31.031 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados2 2024-01-21T08:03:31.176 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:31.223 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:31.224 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:03:31.224 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:31.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rados 2024-01-21T08:03:31.787 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:17 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:32.218 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:21 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:33.323 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:33.373 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:33.374 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:03:33.375 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:33.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librbd1 2024-01-21T08:03:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:33.692 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:03:33.692 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:33.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rgw 2024-01-21T08:03:34.239 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:20 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:34.528 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:35.788 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:35.837 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:35.838 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:03:35.839 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:35.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rados 2024-01-21T08:03:35.965 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:36.010 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:36.012 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:03:36.012 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:36.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-cephfs 2024-01-21T08:03:36.714 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:22 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:38.269 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:38.281 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:38.315 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:03:38.315 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:38.330 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:38.332 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:03:38.332 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:38.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rbd 2024-01-21T08:03:38.435 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rgw 2024-01-21T08:03:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:28 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:39.196 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:40.574 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:40.619 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:40.620 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-01-21T08:03:40.620 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:40.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-fuse 2024-01-21T08:03:40.731 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:40.780 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:40.781 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:03:40.781 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:40.885 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-cephfs 2024-01-21T08:03:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:30 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:41.662 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:27 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:42.915 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:42.915 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-01-21T08:03:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:42.917 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 89 k 2024-01-21T08:03:42.917 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 298 k 2024-01-21T08:03:42.917 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:03:42.961 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-01-21T08:03:42.961 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:42.961 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-01-21T08:03:42.962 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:03:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:03:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:03:43.021 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:03:43.024 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:03:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:03:43.209 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:43.259 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:43.260 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:03:43.260 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:43.369 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rbd 2024-01-21T08:03:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:44.100 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:44.100 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:44.129 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:30 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:44.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-mirror 2024-01-21T08:03:45.050 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:34 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:45.672 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T08:03:45.720 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:45.722 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-21T08:03:45.722 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:45.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-fuse 2024-01-21T08:03:46.549 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:03:46.550 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:46.551 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-01-21T08:03:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:46.551 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 3.7 M 2024-01-21T08:03:46.551 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 18 M 2024-01-21T08:03:46.551 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:03:46.631 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:32 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:46.639 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-01-21T08:03:46.640 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:46.640 INFO:teuthology.orchestra.run.smithi078.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-01-21T08:03:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:03:46.652 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:03:46.652 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:03:46.714 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:03:46.717 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:03:47.182 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:03:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:47.812 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:48.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:48.216 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:48.216 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:48.216 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:48.216 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:48.216 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:48.220 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-21T08:03:48.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:48.222 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 89 k 2024-01-21T08:03:48.222 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 298 k 2024-01-21T08:03:48.222 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:48.283 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.4 MB/s | 89 kB 00:00 2024-01-21T08:03:48.283 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:48.283 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.4 MB/s | 89 kB 00:00 2024-01-21T08:03:48.284 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:48.294 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:48.294 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:48.347 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:48.350 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:48.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-nbd 2024-01-21T08:03:48.454 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:48.680 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:48.985 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:49.154 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:38 ago on Sun 21 Jan 2024 08:01:11 AM UTC. 2024-01-21T08:03:49.379 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:49.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:49.379 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:49.379 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:49.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:49.379 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:49.621 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-mirror 2024-01-21T08:03:50.386 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:36 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:50.638 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-01-21T08:03:50.639 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:50.639 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:50.639 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:50.639 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 173 k 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 649 k 2024-01-21T08:03:50.640 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-01-21T08:03:50.684 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2024-01-21T08:03:50.684 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:50.685 INFO:teuthology.orchestra.run.smithi078.stdout:Total 3.7 MB/s | 173 kB 00:00 2024-01-21T08:03:50.685 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-01-21T08:03:50.695 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-01-21T08:03:50.696 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-01-21T08:03:50.746 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-01-21T08:03:50.749 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-01-21T08:03:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-01-21T08:03:51.096 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:51.380 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-01-21T08:03:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:03:51.783 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-01-21T08:03:51.973 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:51.973 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:51.973 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-21T08:03:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:51.975 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 3.7 M 2024-01-21T08:03:51.975 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 18 M 2024-01-21T08:03:51.975 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:52.013 DEBUG:teuthology.parallel:result is None 2024-01-21T08:03:52.061 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2024-01-21T08:03:52.061 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:52.062 INFO:teuthology.orchestra.run.smithi046.stdout:Total 43 MB/s | 3.7 MB 00:00 2024-01-21T08:03:52.062 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:52.075 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:52.075 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:52.143 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:52.146 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:52.638 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:52.822 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:53.637 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:53.637 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:53.637 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:53.637 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:53.637 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:53.637 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:53.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-nbd 2024-01-21T08:03:54.584 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:40 ago on Sun 21 Jan 2024 08:01:14 AM UTC. 2024-01-21T08:03:56.167 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-21T08:03:56.168 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:56.168 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 173 k 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 649 k 2024-01-21T08:03:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-21T08:03:56.214 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2024-01-21T08:03:56.214 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-21T08:03:56.215 INFO:teuthology.orchestra.run.smithi046.stdout:Total 3.7 MB/s | 173 kB 00:00 2024-01-21T08:03:56.215 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-21T08:03:56.226 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-21T08:03:56.227 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-21T08:03:56.281 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-21T08:03:56.284 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-21T08:03:56.398 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-21T08:03:56.613 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:56.912 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:57.300 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T08:03:57.300 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:57.300 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-21T08:03:57.300 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T08:03:57.300 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:03:57.300 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-21T08:03:57.542 DEBUG:teuthology.parallel:result is None 2024-01-21T08:03:57.542 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T08:03:57.542 INFO:teuthology.packaging:ref: None 2024-01-21T08:03:57.542 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T08:03:57.542 INFO:teuthology.packaging:branch: None 2024-01-21T08:03:57.543 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T08:03:57.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T08:03:57.687 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T08:03:57.710 INFO:teuthology.orchestra.run.smithi046.stdout:16.2.4-0.el8 2024-01-21T08:03:57.710 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T08:03:57.711 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T08:03:57.712 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T08:03:57.712 INFO:teuthology.packaging:ref: None 2024-01-21T08:03:57.712 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T08:03:57.713 INFO:teuthology.packaging:branch: None 2024-01-21T08:03:57.713 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T08:03:57.713 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T08:03:57.933 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T08:03:57.960 INFO:teuthology.orchestra.run.smithi078.stdout:16.2.4-0.el8 2024-01-21T08:03:57.961 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T08:03:57.961 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T08:03:57.963 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T08:03:57.963 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:03:57.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T08:03:57.992 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:03:57.992 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T08:03:58.035 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T08:03:58.035 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:03:58.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T08:03:58.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T08:03:58.131 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:03:58.131 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T08:03:58.165 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T08:03:58.238 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T08:03:58.238 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:03:58.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T08:03:58.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T08:03:58.334 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:03:58.334 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T08:03:58.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T08:03:58.439 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T08:03:58.439 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:03:58.440 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T08:03:58.467 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T08:03:58.535 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:03:58.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T08:03:58.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T08:03:58.638 INFO:teuthology.run_tasks:Running task print... 2024-01-21T08:03:58.649 INFO:teuthology.task.print:**** done install task... 2024-01-21T08:03:58.649 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-21T08:03:58.721 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T08:03:58.721 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-21T08:03:58.722 INFO:tasks.cephadm:Cluster fsid is a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:03:58.722 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-21T08:03:58.722 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-21T08:03:58.722 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi046': '172.21.15.46', 'mon.smithi078': '172.21.15.78'} 2024-01-21T08:03:58.723 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-21T08:03:58.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2024-01-21T08:03:58.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2024-01-21T08:03:58.783 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-21T08:03:58.783 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T08:03:59.002 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 08:03 /home/ubuntu/cephtest/cephadm 2024-01-21T08:03:59.002 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T08:03:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 08:03 /home/ubuntu/cephtest/cephadm 2024-01-21T08:03:59.170 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T08:03:59.193 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T08:03:59.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2024-01-21T08:03:59.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2024-01-21T08:03:59.296 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2024-01-21T08:03:59.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2024-01-21T08:03:59.368 INFO:tasks.cephadm:Writing seed config... 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-21T08:03:59.369 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-21T08:03:59.370 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-21T08:03:59.371 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-21T08:03:59.372 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:03:59.372 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-21T08:03:59.393 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a17ce476-b833-11ee-95b0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-21T08:03:59.393 DEBUG:teuthology.orchestra.run.smithi046:mon.smithi046> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046.service 2024-01-21T08:03:59.436 INFO:tasks.cephadm:Bootstrapping... 2024-01-21T08:03:59.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid a17ce476-b833-11ee-95b0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.46 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-21T08:03:59.465 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:-- Logs begin at Sun 2024-01-21 07:53:41 UTC. -- 2024-01-21T08:03:59.706 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T08:04:00.617 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: 4.6.1 2024-01-21T08:04:00.626 INFO:teuthology.orchestra.run.smithi046.stderr:Verifying podman|docker is present... 2024-01-21T08:04:00.626 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T08:04:00.650 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: 4.6.1 2024-01-21T08:04:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:Verifying lvm2 is present... 2024-01-21T08:04:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:Verifying time synchronization is in place... 2024-01-21T08:04:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T08:04:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T08:04:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active chrony.service 2024-01-21T08:04:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: inactive 2024-01-21T08:04:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T08:04:00.687 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: enabled 2024-01-21T08:04:00.687 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T08:04:00.693 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: active 2024-01-21T08:04:00.694 INFO:teuthology.orchestra.run.smithi046.stderr:Unit chronyd.service is enabled and running 2024-01-21T08:04:00.694 INFO:teuthology.orchestra.run.smithi046.stderr:Repeating the final host check... 2024-01-21T08:04:00.694 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T08:04:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: 4.6.1 2024-01-21T08:04:00.726 INFO:teuthology.orchestra.run.smithi046.stderr:podman|docker (/bin/podman) is present 2024-01-21T08:04:00.726 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl is present 2024-01-21T08:04:00.726 INFO:teuthology.orchestra.run.smithi046.stderr:lvcreate is present 2024-01-21T08:04:00.726 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T08:04:00.733 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T08:04:00.733 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active chrony.service 2024-01-21T08:04:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: inactive 2024-01-21T08:04:00.740 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T08:04:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: enabled 2024-01-21T08:04:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T08:04:00.753 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: active 2024-01-21T08:04:00.753 INFO:teuthology.orchestra.run.smithi046.stderr:Unit chronyd.service is enabled and running 2024-01-21T08:04:00.753 INFO:teuthology.orchestra.run.smithi046.stderr:Host looks OK 2024-01-21T08:04:00.753 INFO:teuthology.orchestra.run.smithi046.stderr:Cluster fsid: a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:00.753 INFO:teuthology.orchestra.run.smithi046.stderr:Acquiring lock 140495455857128 on /run/cephadm/a17ce476-b833-11ee-95b0-87774f69a715.lock 2024-01-21T08:04:00.753 INFO:teuthology.orchestra.run.smithi046.stderr:Lock 140495455857128 acquired on /run/cephadm/a17ce476-b833-11ee-95b0-87774f69a715.lock 2024-01-21T08:04:00.754 INFO:teuthology.orchestra.run.smithi046.stderr:Verifying IP 172.21.15.46 port 3300 ... 2024-01-21T08:04:00.754 INFO:teuthology.orchestra.run.smithi046.stderr:Verifying IP 172.21.15.46 port 6789 ... 2024-01-21T08:04:00.754 INFO:teuthology.orchestra.run.smithi046.stderr:Base mon IP is 172.21.15.46, final addrv is [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-01-21T08:04:00.754 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /sbin/ip route ls 2024-01-21T08:04:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.46 metric 100 2024-01-21T08:04:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.46 metric 100 2024-01-21T08:04:00.757 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /sbin/ip -6 route ls 2024-01-21T08:04:00.759 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-21T08:04:00.759 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-01-21T08:04:00.759 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1791sec hoplimit 64 pref medium 2024-01-21T08:04:00.760 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /sbin/ip -6 addr ls 2024-01-21T08:04:00.762 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-21T08:04:00.762 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-21T08:04:00.762 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T08:04:00.763 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-01-21T08:04:00.763 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-21T08:04:00.763 INFO:teuthology.orchestra.run.smithi046.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T08:04:00.763 INFO:teuthology.orchestra.run.smithi046.stderr:Mon IP 172.21.15.46 is in CIDR network 172.21.0.0/20 2024-01-21T08:04:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-21T08:04:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-21T08:04:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-21T08:04:00.862 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-21T08:04:07.481 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: Getting image source signatures 2024-01-21T08:04:07.491 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-21T08:04:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-21T08:04:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T08:04:20.490 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: Writing manifest to image destination 2024-01-21T08:04:20.760 INFO:teuthology.orchestra.run.smithi046.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T08:04:20.772 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-21T08:04:21.285 INFO:teuthology.orchestra.run.smithi046.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T08:04:21.641 INFO:teuthology.orchestra.run.smithi046.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T08:04:21.641 INFO:teuthology.orchestra.run.smithi046.stderr:Extracting ceph user uid/gid from container image... 2024-01-21T08:04:21.641 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-21T08:04:21.932 INFO:teuthology.orchestra.run.smithi046.stderr:stat: 167 167 2024-01-21T08:04:22.254 INFO:teuthology.orchestra.run.smithi046.stderr:Creating initial keys... 2024-01-21T08:04:22.254 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T08:04:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-authtool: AQAG0KxlKPctIBAAmPR0bfORof0Ap9uidy7s6A== 2024-01-21T08:04:22.883 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T08:04:23.146 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-authtool: AQAH0KxlohadBxAANB5chledZk1aQM5cCBR4cw== 2024-01-21T08:04:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T08:04:23.754 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-authtool: AQAH0Kxl6G5dLBAAHERxKv3CKNOJE52/QLMDYA== 2024-01-21T08:04:24.082 INFO:teuthology.orchestra.run.smithi046.stderr:Creating initial monmap... 2024-01-21T08:04:24.082 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmph5vm33h3:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid a17ce476-b833-11ee-95b0-87774f69a715 --addv smithi046 [v2:172.21.15.46:3300,v1:172.21.15.46:6789] /tmp/monmap 2024-01-21T08:04:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T08:04:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T08:04:24.678 INFO:teuthology.orchestra.run.smithi046.stderr:monmaptool for smithi046 [v2:172.21.15.46:3300,v1:172.21.15.46:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T08:04:24.678 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/monmaptool: set fsid to a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:24.678 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T08:04:24.679 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:04:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:Creating mon... 2024-01-21T08:04:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046:/var/lib/ceph/mon/ceph-smithi046:z -v /tmp/ceph-tmpx4zy1xdy:/tmp/keyring:z -v /tmp/ceph-tmph5vm33h3:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi046 --fsid a17ce476-b833-11ee-95b0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.931+0000 7f942b93f700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.931+0000 7f942b93f700 1 imported monmap: 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: fsid a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: last_changed 2024-01-21T08:04:24.344932+0000 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: created 2024-01-21T08:04:24.344932+0000 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-21T08:04:24.972 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.smithi046 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.931+0000 7f942b93f700 0 /usr/bin/ceph-mon: set fsid to a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Compile date May 13 2021 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: DB SUMMARY 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi046/store.db dir, Total Num: 0, files: 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi046/store.db: 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T08:04:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.create_if_missing: 1 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.env: 0x55784b4a51c0 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.fs: Posix File System 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.info_log: 0x55784dbe0ec0 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.statistics: (nil) 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.use_fsync: 0 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T08:04:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.db_log_dir: 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi046/store.db 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T08:04:24.975 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.write_buffer_manager: 0x55784dbe7530 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.unordered_write: 0 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T08:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.row_cache: None 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.wal_filter: None 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T08:04:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.max_open_files: -1 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Compression algorithms supported: 2024-01-21T08:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kZSTD supported: 0 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.932+0000 7f942b93f700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi046/store.db/MANIFEST-000001 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T08:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.merge_operator: 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_filter: None 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55784daf8120) 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-21T08:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_cache: 0x55784db2ed10 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-21T08:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-21T08:04:24.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compression: NoCompression 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.num_levels: 7 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T08:04:24.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T08:04:24.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T08:04:24.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T08:04:24.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.ttl: 2592000 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi046/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.933+0000 7f942b93f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.934+0000 7f942b93f700 4 rocksdb: DB pointer 0x55784dbf3800 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.934+0000 7f94147d8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.934+0000 7f94147d8700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T08:04:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:24.989 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T08:04:24.990 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T08:04:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T08:04:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: 2024-01-21T08:04:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.935+0000 7f942b93f700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-21T08:04:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.935+0000 7f942b93f700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-21T08:04:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph-mon: debug 2024-01-21T08:04:24.935+0000 7f942b93f700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi046 for mon.smithi046 2024-01-21T08:04:25.271 INFO:teuthology.orchestra.run.smithi046.stderr:create mon.smithi046 on 2024-01-21T08:04:25.271 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:25.276 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl disable ceph.target 2024-01-21T08:04:25.283 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-21T08:04:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl enable ceph.target 2024-01-21T08:04:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-21T08:04:25.464 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl start ceph.target 2024-01-21T08:04:25.471 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl enable ceph-a17ce476-b833-11ee-95b0-87774f69a715.target 2024-01-21T08:04:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a17ce476-b833-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-a17ce476-b833-11ee-95b0-87774f69a715.target. 2024-01-21T08:04:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-a17ce476-b833-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-a17ce476-b833-11ee-95b0-87774f69a715.target. 2024-01-21T08:04:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715.target 2024-01-21T08:04:25.569 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl daemon-reload 2024-01-21T08:04:25.658 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046 2024-01-21T08:04:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl reset-failed ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046 2024-01-21T08:04:25.691 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Failed to reset failed state of unit ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046.service: Unit ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046.service not loaded. 2024-01-21T08:04:25.691 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl enable ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046 2024-01-21T08:04:25.698 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a17ce476-b833-11ee-95b0-87774f69a715.target.wants/ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046.service → /etc/systemd/system/ceph-a17ce476-b833-11ee-95b0-87774f69a715@.service. 2024-01-21T08:04:25.785 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046 2024-01-21T08:04:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T08:04:26.101 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: disabled 2024-01-21T08:04:26.101 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T08:04:26.109 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: inactive 2024-01-21T08:04:26.110 INFO:teuthology.orchestra.run.smithi046.stderr:firewalld.service is not enabled 2024-01-21T08:04:26.110 INFO:teuthology.orchestra.run.smithi046.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T08:04:26.110 INFO:teuthology.orchestra.run.smithi046.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T08:04:26.111 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for mon to start... 2024-01-21T08:04:26.111 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for mon... 2024-01-21T08:04:26.111 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046:/var/lib/ceph/mon/ceph-smithi046:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-21T08:04:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.468+0000 7fcebd683700 1 Processor -- start 2024-01-21T08:04:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.468+0000 7fcebd683700 1 -- start start 2024-01-21T08:04:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.468+0000 7fcebd683700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.468+0000 7fcebd683700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80fb490 con 0x7fceb80fab80 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.469+0000 7fceb6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.469+0000 7fceb6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51552/0 (socket says 172.21.15.46:51552) 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.469+0000 7fceb6ffd700 1 -- 172.21.15.46:0/3122359120 learned_addr learned my addr 172.21.15.46:0/3122359120 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.469+0000 7fceb6ffd700 1 -- 172.21.15.46:0/3122359120 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb80fb5d0 con 0x7fceb80fab80 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.469+0000 7fceb6ffd700 1 --2- 172.21.15.46:0/3122359120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fcea800b5f0 tx=0x7fcea8006ee0).ready entity=mon.0 client_cookie=de69590a3307912a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.469+0000 7fceb5ffb700 1 -- 172.21.15.46:0/3122359120 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea800d600 con 0x7fceb80fab80 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fceb5ffb700 1 -- 172.21.15.46:0/3122359120 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea800d760 con 0x7fceb80fab80 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fceb5ffb700 1 -- 172.21.15.46:0/3122359120 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea800da30 con 0x7fceb80fab80 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fcebd683700 1 -- 172.21.15.46:0/3122359120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 msgr2=0x7fceb80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fcebd683700 1 --2- 172.21.15.46:0/3122359120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fcea800b5f0 tx=0x7fcea8006ee0).stop 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fcebd683700 1 -- 172.21.15.46:0/3122359120 shutdown_connections 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fcebd683700 1 --2- 172.21.15.46:0/3122359120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fcebd683700 1 -- 172.21.15.46:0/3122359120 >> 172.21.15.46:0/3122359120 conn(0x7fceb80f63c0 msgr2=0x7fceb80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fcebd683700 1 -- 172.21.15.46:0/3122359120 shutdown_connections 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.470+0000 7fcebd683700 1 -- 172.21.15.46:0/3122359120 wait complete. 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.471+0000 7fcebd683700 1 Processor -- start 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.471+0000 7fcebd683700 1 -- start start 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.471+0000 7fcebd683700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.471+0000 7fcebd683700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80fd200 con 0x7fceb80fab80 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.471+0000 7fceb6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.471+0000 7fceb6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51562/0 (socket says 172.21.15.46:51562) 2024-01-21T08:04:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.471+0000 7fceb6ffd700 1 -- 172.21.15.46:0/1260493803 learned_addr learned my addr 172.21.15.46:0/1260493803 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fceb6ffd700 1 -- 172.21.15.46:0/1260493803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea800b200 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fceb6ffd700 1 --2- 172.21.15.46:0/1260493803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fed80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcea8006c70 tx=0x7fcea8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fceaf7fe700 1 -- 172.21.15.46:0/1260493803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea8011b40 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fceaf7fe700 1 -- 172.21.15.46:0/1260493803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea8011ca0 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fceaf7fe700 1 -- 172.21.15.46:0/1260493803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea801d430 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb80fd3a0 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb80fd7b0 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fceaf7fe700 1 -- 172.21.15.46:0/1260493803 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fcea800f070 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.472+0000 7fceaf7fe700 1 -- 172.21.15.46:0/1260493803 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcea8027460 con 0x7fceb80fab80 2024-01-21T08:04:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.473+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb8059100 con 0x7fceb80fab80 2024-01-21T08:04:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.476+0000 7fceaf7fe700 1 -- 172.21.15.46:0/1260493803 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fcea8016070 con 0x7fceb80fab80 2024-01-21T08:04:26.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: cluster 2024-01-21T08:04:26.107449+0000 mon.smithi046 (mon.0) 1 : cluster [INF] mon.smithi046 is new leader, mons smithi046 in quorum (ranks 0) 2024-01-21T08:04:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.544+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fceb8103c80 con 0x7fceb80fab80 2024-01-21T08:04:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.545+0000 7fceaf7fe700 1 -- 172.21.15.46:0/1260493803 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fcea8035160 con 0x7fceb80fab80 2024-01-21T08:04:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: cluster: 2024-01-21T08:04:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: id: a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-21T08:04:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: services: 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi046 (age 0.423678s) 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: data: 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-21T08:04:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: pgs: 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.545+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 msgr2=0x7fceb80fed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.545+0000 7fcebd683700 1 --2- 172.21.15.46:0/1260493803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fed80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcea8006c70 tx=0x7fcea8016000).stop 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.545+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 shutdown_connections 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.545+0000 7fcebd683700 1 --2- 172.21.15.46:0/1260493803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80fab80 0x7fceb80fed80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.545+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 >> 172.21.15.46:0/1260493803 conn(0x7fceb80f63c0 msgr2=0x7fceb806ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.546+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 shutdown_connections 2024-01-21T08:04:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:26.546+0000 7fcebd683700 1 -- 172.21.15.46:0/1260493803 wait complete. 2024-01-21T08:04:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:mon is available 2024-01-21T08:04:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:Assimilating anything we can from ceph.conf... 2024-01-21T08:04:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046:/var/lib/ceph/mon/ceph-smithi046:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi046/config 2024-01-21T08:04:27.206 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.200+0000 7f7e90c22700 1 Processor -- start 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.200+0000 7f7e90c22700 1 -- start start 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.201+0000 7f7e90c22700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.201+0000 7f7e90c22700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0fb490 con 0x7f7e8c0fab80 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.201+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.201+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51576/0 (socket says 172.21.15.46:51576) 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.201+0000 7f7e8a59c700 1 -- 172.21.15.46:0/2454705714 learned_addr learned my addr 172.21.15.46:0/2454705714 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.201+0000 7f7e8a59c700 1 -- 172.21.15.46:0/2454705714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c0fb5d0 con 0x7f7e8c0fab80 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.201+0000 7f7e8a59c700 1 --2- 172.21.15.46:0/2454705714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c0fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f7e7c00b5f0 tx=0x7f7e7c006ee0).ready entity=mon.0 client_cookie=901d4646a36ec33f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e8959a700 1 -- 172.21.15.46:0/2454705714 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7c00d600 con 0x7f7e8c0fab80 2024-01-21T08:04:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e8959a700 1 -- 172.21.15.46:0/2454705714 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c00d760 con 0x7f7e8c0fab80 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e8959a700 1 -- 172.21.15.46:0/2454705714 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7c00da30 con 0x7f7e8c0fab80 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e90c22700 1 -- 172.21.15.46:0/2454705714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 msgr2=0x7f7e8c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e90c22700 1 --2- 172.21.15.46:0/2454705714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c0fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f7e7c00b5f0 tx=0x7f7e7c006ee0).stop 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e90c22700 1 -- 172.21.15.46:0/2454705714 shutdown_connections 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e90c22700 1 --2- 172.21.15.46:0/2454705714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c0fafa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e90c22700 1 -- 172.21.15.46:0/2454705714 >> 172.21.15.46:0/2454705714 conn(0x7f7e8c0f63c0 msgr2=0x7f7e8c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e90c22700 1 -- 172.21.15.46:0/2454705714 shutdown_connections 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.202+0000 7f7e90c22700 1 -- 172.21.15.46:0/2454705714 wait complete. 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e90c22700 1 Processor -- start 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e90c22700 1 -- start start 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e90c22700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c10c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:27.208 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e90c22700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c10c950 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c10c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c10c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51592/0 (socket says 172.21.15.46:51592) 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e8a59c700 1 -- 172.21.15.46:0/3027411725 learned_addr learned my addr 172.21.15.46:0/3027411725 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.203+0000 7f7e8a59c700 1 -- 172.21.15.46:0/3027411725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c00b200 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e8a59c700 1 --2- 172.21.15.46:0/3027411725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c10c460 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f7e7c006c70 tx=0x7f7e7c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7c011b40 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c011ca0 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7c01d430 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c10cb50 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c10cf60 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f7e7c00f070 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.204+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7e7c027460 con 0x7f7e8c0fab80 2024-01-21T08:04:27.209 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.205+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c045ad0 con 0x7f7e8c0fab80 2024-01-21T08:04:27.210 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.207+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7e7c016070 con 0x7f7e8c0fab80 2024-01-21T08:04:27.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: cluster 2024-01-21T08:04:26.122103+0000 mon.smithi046 (mon.0) 2 : cluster [INF] mon.smithi046 is new leader, mons smithi046 in quorum (ranks 0) 2024-01-21T08:04:27.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: cluster 2024-01-21T08:04:26.122309+0000 mon.smithi046 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi046=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} 2024-01-21T08:04:27.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: cluster 2024-01-21T08:04:26.124089+0000 mon.smithi046 (mon.0) 4 : cluster [DBG] fsmap 2024-01-21T08:04:27.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: cluster 2024-01-21T08:04:26.126094+0000 mon.smithi046 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T08:04:27.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: cluster 2024-01-21T08:04:26.126455+0000 mon.smithi046 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T08:04:27.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: audit 2024-01-21T08:04:26.545737+0000 mon.smithi046 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.46:0/1260493803' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-21T08:04:27.276 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.274+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7e8c0fbec0 con 0x7f7e8c0fab80 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.278+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7e7c01d620 con 0x7f7e8c0fab80 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.278+0000 7f7e82ffd700 1 -- 172.21.15.46:0/3027411725 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f7e7c0308f0 con 0x7f7e8c0fab80 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: [global] 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: fsid = a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-21T08:04:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: [mgr] 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: [osd] 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.279+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 msgr2=0x7f7e8c10c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.279+0000 7f7e90c22700 1 --2- 172.21.15.46:0/3027411725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c10c460 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f7e7c006c70 tx=0x7f7e7c016000).stop 2024-01-21T08:04:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.279+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 shutdown_connections 2024-01-21T08:04:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.279+0000 7f7e90c22700 1 --2- 172.21.15.46:0/3027411725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8c0fab80 0x7f7e8c10c460 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.279+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 >> 172.21.15.46:0/3027411725 conn(0x7f7e8c0f63c0 msgr2=0x7f7e8c0f85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.279+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 shutdown_connections 2024-01-21T08:04:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.279+0000 7f7e90c22700 1 -- 172.21.15.46:0/3027411725 wait complete. 2024-01-21T08:04:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:Generating new minimal ceph.conf... 2024-01-21T08:04:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046:/var/lib/ceph/mon/ceph-smithi046:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi046/config 2024-01-21T08:04:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.999+0000 7f74e1f05700 1 Processor -- start 2024-01-21T08:04:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.999+0000 7f74e1f05700 1 -- start start 2024-01-21T08:04:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.999+0000 7f74e1f05700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:27.999+0000 7f74e1f05700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0fb490 con 0x7f74dc0fab80 2024-01-21T08:04:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51604/0 (socket says 172.21.15.46:51604) 2024-01-21T08:04:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74db7fe700 1 -- 172.21.15.46:0/1985730125 learned_addr learned my addr 172.21.15.46:0/1985730125 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74db7fe700 1 -- 172.21.15.46:0/1985730125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0fb5d0 con 0x7f74dc0fab80 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74db7fe700 1 --2- 172.21.15.46:0/1985730125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc0fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f74d000b5f0 tx=0x7f74d0006ee0).ready entity=mon.0 client_cookie=a58e79b04376f3e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74da7fc700 1 -- 172.21.15.46:0/1985730125 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d000d600 con 0x7f74dc0fab80 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74da7fc700 1 -- 172.21.15.46:0/1985730125 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f74d000dbd0 con 0x7f74dc0fab80 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.000+0000 7f74da7fc700 1 -- 172.21.15.46:0/1985730125 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d0011c00 con 0x7f74dc0fab80 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 -- 172.21.15.46:0/1985730125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 msgr2=0x7f74dc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 --2- 172.21.15.46:0/1985730125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc0fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f74d000b5f0 tx=0x7f74d0006ee0).stop 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 -- 172.21.15.46:0/1985730125 shutdown_connections 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 --2- 172.21.15.46:0/1985730125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc0fafa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 -- 172.21.15.46:0/1985730125 >> 172.21.15.46:0/1985730125 conn(0x7f74dc0f63c0 msgr2=0x7f74dc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 -- 172.21.15.46:0/1985730125 shutdown_connections 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 -- 172.21.15.46:0/1985730125 wait complete. 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.001+0000 7f74e1f05700 1 Processor -- start 2024-01-21T08:04:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74e1f05700 1 -- start start 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74e1f05700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc108810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74e1f05700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc108d00 con 0x7f74dc0fab80 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc108810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc108810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51612/0 (socket says 172.21.15.46:51612) 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74db7fe700 1 -- 172.21.15.46:0/1344738452 learned_addr learned my addr 172.21.15.46:0/1344738452 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74db7fe700 1 -- 172.21.15.46:0/1344738452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d000b200 con 0x7f74dc0fab80 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74db7fe700 1 --2- 172.21.15.46:0/1344738452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc108810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f74d000e040 tx=0x7f74d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74c3fff700 1 -- 172.21.15.46:0/1344738452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d001b410 con 0x7f74dc0fab80 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74c3fff700 1 -- 172.21.15.46:0/1344738452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f74d001b9e0 con 0x7f74dc0fab80 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc108f00 con 0x7f74dc0fab80 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.002+0000 7f74c3fff700 1 -- 172.21.15.46:0/1344738452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d001a820 con 0x7f74dc0fab80 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.003+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc109310 con 0x7f74dc0fab80 2024-01-21T08:04:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.003+0000 7f74c3fff700 1 -- 172.21.15.46:0/1344738452 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f74d000f070 con 0x7f74dc0fab80 2024-01-21T08:04:28.009 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.003+0000 7f74c3fff700 1 -- 172.21.15.46:0/1344738452 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f74d0025b40 con 0x7f74dc0fab80 2024-01-21T08:04:28.009 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.004+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc045ad0 con 0x7f74dc0fab80 2024-01-21T08:04:28.009 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.006+0000 7f74c3fff700 1 -- 172.21.15.46:0/1344738452 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f74d0016070 con 0x7f74dc0fab80 2024-01-21T08:04:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.073+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f74dc102670 con 0x7f74dc0fab80 2024-01-21T08:04:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.073+0000 7f74c3fff700 1 -- 172.21.15.46:0/1344738452 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f74d0025dd0 con 0x7f74dc0fab80 2024-01-21T08:04:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.074+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 msgr2=0x7f74dc108810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.074+0000 7f74e1f05700 1 --2- 172.21.15.46:0/1344738452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc108810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f74d000e040 tx=0x7f74d0006ee0).stop 2024-01-21T08:04:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.074+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 shutdown_connections 2024-01-21T08:04:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.074+0000 7f74e1f05700 1 --2- 172.21.15.46:0/1344738452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74dc0fab80 0x7f74dc108810 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.074+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 >> 172.21.15.46:0/1344738452 conn(0x7f74dc0f63c0 msgr2=0x7f74dc0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.074+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 shutdown_connections 2024-01-21T08:04:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:28.074+0000 7f74e1f05700 1 -- 172.21.15.46:0/1344738452 wait complete. 2024-01-21T08:04:28.416 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: audit 2024-01-21T08:04:27.275638+0000 mon.smithi046 (mon.0) 8 : audit [INF] from='client.? 172.21.15.46:0/3027411725' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T08:04:28.416 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: audit 2024-01-21T08:04:27.278701+0000 mon.smithi046 (mon.0) 9 : audit [INF] from='client.? 172.21.15.46:0/3027411725' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T08:04:28.416 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: audit 2024-01-21T08:04:28.073970+0000 mon.smithi046 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.46:0/1344738452' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:04:28.453 INFO:teuthology.orchestra.run.smithi046.stderr:Restarting the monitor... 2024-01-21T08:04:28.453 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046 2024-01-21T08:04:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 systemd[1]: Stopping Ceph mon.smithi046 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:04:28.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: debug 2024-01-21T08:04:28.527+0000 7fd627e7e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi046 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T08:04:28.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38396]: debug 2024-01-21T08:04:28.527+0000 7fd627e7e700 -1 mon.smithi046@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T08:04:29.165 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 podman[38718]: ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046 2024-01-21T08:04:29.166 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 systemd[1]: ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046.service: Succeeded. 2024-01-21T08:04:29.166 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 systemd[1]: Stopped Ceph mon.smithi046 for a17ce476-b833-11ee-95b0-87774f69a715. 2024-01-21T08:04:29.166 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:28 smithi046 systemd[1]: Starting Ceph mon.smithi046 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:04:29.240 INFO:teuthology.orchestra.run.smithi046.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-21T08:04:29.241 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.229+0000 7f598cb5e700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.229+0000 7f598cb5e700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.229+0000 7f598cb5e700 0 pidfile_write: ignore empty --pid-file 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.233+0000 7f598cb5e700 0 load: jerasure load: lrc load: isa 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Compile date May 13 2021 2024-01-21T08:04:29.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: DB SUMMARY 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi046/store.db dir, Total Num: 1, files: 000004.sst 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi046/store.db: 000006.log size: 93438 ; 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T08:04:29.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 bash[38790]: 5272fc393ba8f5e391c886ddf9907c5e49ba6033009d079aac63ad3adcfc978a 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.env: 0x557b23fc01c0 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.fs: Posix File System 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.info_log: 0x557b24d4d960 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.statistics: (nil) 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.use_fsync: 0 2024-01-21T08:04:29.495 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.db_log_dir: 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi046/store.db 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T08:04:29.496 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.write_buffer_manager: 0x557b259c2030 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T08:04:29.497 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.unordered_write: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.row_cache: None 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.wal_filter: None 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T08:04:29.498 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T08:04:29.499 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_open_files: -1 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Compression algorithms supported: 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kZSTD supported: 0 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T08:04:29.500 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi046/store.db/MANIFEST-000005 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T08:04:29.501 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.merge_operator: 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_filter: None 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557b24c64128) 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cache_index_and_filter_blocks: 1 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: pin_top_level_index_and_filter: 1 2024-01-21T08:04:29.502 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: index_type: 0 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: data_block_index_type: 0 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: index_shortening: 1 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: hash_index_allow_collision: 1 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: checksum: 1 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: no_block_cache: 0 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_cache: 0x557b24c9af10 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_cache_name: BinnedLRUCache 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_cache_options: 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: capacity : 536870912 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: num_shard_bits : 4 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: strict_capacity_limit : 0 2024-01-21T08:04:29.503 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: high_pri_pool_ratio: 0.000 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_cache_compressed: (nil) 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: persistent_cache: (nil) 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_size: 4096 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_size_deviation: 10 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_restart_interval: 16 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: index_block_restart_interval: 1 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: metadata_block_size: 4096 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: partition_filters: 0 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: use_delta_encoding: 1 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: whole_key_filtering: 1 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: verify_compression: 0 2024-01-21T08:04:29.504 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: read_amp_bytes_per_bit: 0 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: format_version: 2 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: enable_index_compression: 1 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: block_align: 0 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compression: NoCompression 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.num_levels: 7 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T08:04:29.505 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T08:04:29.506 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T08:04:29.507 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T08:04:29.508 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T08:04:29.509 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.ttl: 2592000 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.234+0000 7f598cb5e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.235+0000 7f598cb5e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi046/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.235+0000 7f598cb5e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.235+0000 7f598cb5e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705824269236044, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.235+0000 7f598cb5e700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.235+0000 7f598cb5e700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.236+0000 7f598cb5e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705824269236993, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 89218, "table_properties": {"data_size": 87333, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 80474, "raw_average_value_size": 323, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705824269, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.236+0000 7f598cb5e700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.236+0000 7f598cb5e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705824269237405, "job": 1, "event": "recovery_finished"} 2024-01-21T08:04:29.510 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.237+0000 7f598cb5e700 4 rocksdb: DB pointer 0x557b24d5f800 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.237+0000 7f5973976700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.237+0000 7f5973976700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ** DB Stats ** 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T08:04:29.511 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ** Compaction Stats [default] ** 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: L0 2/0 88.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Sum 2/0 88.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ** Compaction Stats [default] ** 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:29.512 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.8 0.00 0.00 1 0.001 0 0 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Cumulative compaction: 0.00 GB write, 28.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Interval compaction: 0.00 GB write, 28.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T08:04:29.513 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ** Compaction Stats [default] ** 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: L0 2/0 88.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Sum 2/0 88.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ** Compaction Stats [default] ** 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.8 0.00 0.00 1 0.001 0 0 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:29.514 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Cumulative compaction: 0.00 GB write, 28.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.515 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 systemd[1]: Started Ceph mon.smithi046 for a17ce476-b833-11ee-95b0-87774f69a715. 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.238+0000 7f598cb5e700 0 starting mon.smithi046 rank 0 at public addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] at bind addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi046 fsid a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 1 mon.smithi046@-1(???) e1 preinit fsid a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 0 mon.smithi046@-1(???).mds e1 new map 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 0 mon.smithi046@-1(???).mds e1 print_map 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: e1 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: legacy client fscid: -1 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: No filesystems configured 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 0 mon.smithi046@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 0 mon.smithi046@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T08:04:29.516 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 0 mon.smithi046@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 0 mon.smithi046@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:27.275638+0000 mon.smithi046 (mon.0) 8 : audit [INF] from='client.? 172.21.15.46:0/3027411725' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:27.278701+0000 mon.smithi046 (mon.0) 9 : audit [INF] from='client.? 172.21.15.46:0/3027411725' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:28.073970+0000 mon.smithi046 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.46:0/1344738452' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:04:29.239+0000 7f598cb5e700 1 mon.smithi046@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:29.255453+0000 mon.smithi046 (mon.0) 1 : cluster [INF] mon.smithi046 is new leader, mons smithi046 in quorum (ranks 0) 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:29.255546+0000 mon.smithi046 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi046=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:29.255611+0000 mon.smithi046 (mon.0) 3 : cluster [DBG] fsmap 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:29.255659+0000 mon.smithi046 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T08:04:29.517 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:29.256201+0000 mon.smithi046 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.581+0000 7f1130bf2700 1 Processor -- start 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.581+0000 7f1130bf2700 1 -- start start 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.581+0000 7f1130bf2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.581+0000 7f1130bf2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0fc730 con 0x7f112c0703d0 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.582+0000 7f112a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.582+0000 7f112a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51622/0 (socket says 172.21.15.46:51622) 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.582+0000 7f112a59c700 1 -- 172.21.15.46:0/1708206323 learned_addr learned my addr 172.21.15.46:0/1708206323 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.582+0000 7f112a59c700 1 -- 172.21.15.46:0/1708206323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c101230 con 0x7f112c0703d0 2024-01-21T08:04:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.582+0000 7f112a59c700 1 --2- 172.21.15.46:0/1708206323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c100d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f112000b5f0 tx=0x7f1120006ee0).ready entity=mon.0 client_cookie=eeac14700c36ff54 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.582+0000 7f112959a700 1 -- 172.21.15.46:0/1708206323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112000d600 con 0x7f112c0703d0 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f112959a700 1 -- 172.21.15.46:0/1708206323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f112000dbd0 con 0x7f112c0703d0 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f112959a700 1 -- 172.21.15.46:0/1708206323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1120011c00 con 0x7f112c0703d0 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f1130bf2700 1 -- 172.21.15.46:0/1708206323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 msgr2=0x7f112c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f1130bf2700 1 --2- 172.21.15.46:0/1708206323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c100d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f112000b5f0 tx=0x7f1120006ee0).stop 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f1130bf2700 1 -- 172.21.15.46:0/1708206323 shutdown_connections 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f1130bf2700 1 --2- 172.21.15.46:0/1708206323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c100d40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f1130bf2700 1 -- 172.21.15.46:0/1708206323 >> 172.21.15.46:0/1708206323 conn(0x7f112c0f63c0 msgr2=0x7f112c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f1130bf2700 1 -- 172.21.15.46:0/1708206323 shutdown_connections 2024-01-21T08:04:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.583+0000 7f1130bf2700 1 -- 172.21.15.46:0/1708206323 wait complete. 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f1130bf2700 1 Processor -- start 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f1130bf2700 1 -- start start 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f1130bf2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f1130bf2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c10d0a0 con 0x7f112c0703d0 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f112a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f112a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51632/0 (socket says 172.21.15.46:51632) 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f112a59c700 1 -- 172.21.15.46:0/1143268896 learned_addr learned my addr 172.21.15.46:0/1143268896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f112a59c700 1 -- 172.21.15.46:0/1143268896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112000b200 con 0x7f112c0703d0 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.584+0000 7f112a59c700 1 --2- 172.21.15.46:0/1143268896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c10cbb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f112000b190 tx=0x7f1120006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.585+0000 7f111affd700 1 -- 172.21.15.46:0/1143268896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112001b410 con 0x7f112c0703d0 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.585+0000 7f111affd700 1 -- 172.21.15.46:0/1143268896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f112001b9e0 con 0x7f112c0703d0 2024-01-21T08:04:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.585+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c10d2a0 con 0x7f112c0703d0 2024-01-21T08:04:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.585+0000 7f111affd700 1 -- 172.21.15.46:0/1143268896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112001a880 con 0x7f112c0703d0 2024-01-21T08:04:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.585+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c10d6b0 con 0x7f112c0703d0 2024-01-21T08:04:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.585+0000 7f111affd700 1 -- 172.21.15.46:0/1143268896 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f112000f070 con 0x7f112c0703d0 2024-01-21T08:04:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.585+0000 7f111affd700 1 -- 172.21.15.46:0/1143268896 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1120025b70 con 0x7f112c0703d0 2024-01-21T08:04:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.586+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112c045ad0 con 0x7f112c0703d0 2024-01-21T08:04:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.588+0000 7f111affd700 1 -- 172.21.15.46:0/1143268896 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1120016070 con 0x7f112c0703d0 2024-01-21T08:04:29.656 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.654+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f112c101d90 con 0x7f112c0703d0 2024-01-21T08:04:29.658 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.656+0000 7f111affd700 1 -- 172.21.15.46:0/1143268896 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1120025db0 con 0x7f112c0703d0 2024-01-21T08:04:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.657+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 msgr2=0x7f112c10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.657+0000 7f1130bf2700 1 --2- 172.21.15.46:0/1143268896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c10cbb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f112000b190 tx=0x7f1120006ee0).stop 2024-01-21T08:04:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.657+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 shutdown_connections 2024-01-21T08:04:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.657+0000 7f1130bf2700 1 --2- 172.21.15.46:0/1143268896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f112c0703d0 0x7f112c10cbb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.657+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 >> 172.21.15.46:0/1143268896 conn(0x7f112c0f63c0 msgr2=0x7f112c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.657+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 shutdown_connections 2024-01-21T08:04:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:29.657+0000 7f1130bf2700 1 -- 172.21.15.46:0/1143268896 wait complete. 2024-01-21T08:04:30.014 INFO:teuthology.orchestra.run.smithi046.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-21T08:04:30.014 INFO:teuthology.orchestra.run.smithi046.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-21T08:04:30.015 INFO:teuthology.orchestra.run.smithi046.stderr:Creating mgr... 2024-01-21T08:04:30.015 INFO:teuthology.orchestra.run.smithi046.stderr:Verifying port 9283 ... 2024-01-21T08:04:30.016 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl daemon-reload 2024-01-21T08:04:30.116 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@mgr.smithi046.lmxnwc 2024-01-21T08:04:30.142 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl reset-failed ceph-a17ce476-b833-11ee-95b0-87774f69a715@mgr.smithi046.lmxnwc 2024-01-21T08:04:30.148 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Failed to reset failed state of unit ceph-a17ce476-b833-11ee-95b0-87774f69a715@mgr.smithi046.lmxnwc.service: Unit ceph-a17ce476-b833-11ee-95b0-87774f69a715@mgr.smithi046.lmxnwc.service not loaded. 2024-01-21T08:04:30.148 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl enable ceph-a17ce476-b833-11ee-95b0-87774f69a715@mgr.smithi046.lmxnwc 2024-01-21T08:04:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a17ce476-b833-11ee-95b0-87774f69a715.target.wants/ceph-a17ce476-b833-11ee-95b0-87774f69a715@mgr.smithi046.lmxnwc.service → /etc/systemd/system/ceph-a17ce476-b833-11ee-95b0-87774f69a715@.service. 2024-01-21T08:04:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@mgr.smithi046.lmxnwc 2024-01-21T08:04:30.573 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T08:04:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: disabled 2024-01-21T08:04:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T08:04:30.593 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: inactive 2024-01-21T08:04:30.594 INFO:teuthology.orchestra.run.smithi046.stderr:firewalld.service is not enabled 2024-01-21T08:04:30.594 INFO:teuthology.orchestra.run.smithi046.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T08:04:30.594 INFO:teuthology.orchestra.run.smithi046.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T08:04:30.594 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T08:04:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: disabled 2024-01-21T08:04:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T08:04:30.611 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: inactive 2024-01-21T08:04:30.612 INFO:teuthology.orchestra.run.smithi046.stderr:firewalld.service is not enabled 2024-01-21T08:04:30.612 INFO:teuthology.orchestra.run.smithi046.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-21T08:04:30.613 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for mgr to start... 2024-01-21T08:04:30.613 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for mgr... 2024-01-21T08:04:30.613 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T08:04:30.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.978+0000 7fa2cd425700 1 Processor -- start 2024-01-21T08:04:30.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.978+0000 7fa2cd425700 1 -- start start 2024-01-21T08:04:30.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.978+0000 7fa2cd425700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:30.982 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.978+0000 7fa2cd425700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c80fedd0 con 0x7fa2c8070430 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50076/0 (socket says 172.21.15.46:50076) 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c6ffd700 1 -- 172.21.15.46:0/4262668975 learned_addr learned my addr 172.21.15.46:0/4262668975 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c6ffd700 1 -- 172.21.15.46:0/4262668975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c80fef10 con 0x7fa2c8070430 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c6ffd700 1 --2- 172.21.15.46:0/4262668975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa2b800b5f0 tx=0x7fa2b8006ee0).ready entity=mon.0 client_cookie=cdfff172048c84bf server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c5ffb700 1 -- 172.21.15.46:0/4262668975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2b800d600 con 0x7fa2c8070430 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c5ffb700 1 -- 172.21.15.46:0/4262668975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa2b800dbd0 con 0x7fa2c8070430 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.979+0000 7fa2c5ffb700 1 -- 172.21.15.46:0/4262668975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2b8011c00 con 0x7fa2c8070430 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.980+0000 7fa2cd425700 1 -- 172.21.15.46:0/4262668975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 msgr2=0x7fa2c80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.980+0000 7fa2cd425700 1 --2- 172.21.15.46:0/4262668975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa2b800b5f0 tx=0x7fa2b8006ee0).stop 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.980+0000 7fa2cd425700 1 -- 172.21.15.46:0/4262668975 shutdown_connections 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.980+0000 7fa2cd425700 1 --2- 172.21.15.46:0/4262668975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:30.983 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.980+0000 7fa2cd425700 1 -- 172.21.15.46:0/4262668975 >> 172.21.15.46:0/4262668975 conn(0x7fa2c80f63c0 msgr2=0x7fa2c80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:30.984 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.980+0000 7fa2cd425700 1 -- 172.21.15.46:0/4262668975 shutdown_connections 2024-01-21T08:04:30.985 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.980+0000 7fa2cd425700 1 -- 172.21.15.46:0/4262668975 wait complete. 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.981+0000 7fa2cd425700 1 Processor -- start 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.981+0000 7fa2cd425700 1 -- start start 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.981+0000 7fa2cd425700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.981+0000 7fa2cd425700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c80fa7e0 con 0x7fa2c8070430 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50078/0 (socket says 172.21.15.46:50078) 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2c6ffd700 1 -- 172.21.15.46:0/3912295299 learned_addr learned my addr 172.21.15.46:0/3912295299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2c6ffd700 1 -- 172.21.15.46:0/3912295299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b800b200 con 0x7fa2c8070430 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2c6ffd700 1 --2- 172.21.15.46:0/3912295299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa2b800e040 tx=0x7fa2b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2af7fe700 1 -- 172.21.15.46:0/3912295299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2b800d9f0 con 0x7fa2c8070430 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2af7fe700 1 -- 172.21.15.46:0/3912295299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa2b801a9f0 con 0x7fa2c8070430 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c80fa980 con 0x7fa2c8070430 2024-01-21T08:04:30.986 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2af7fe700 1 -- 172.21.15.46:0/3912295299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2b801b8c0 con 0x7fa2c8070430 2024-01-21T08:04:30.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.982+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c80fad90 con 0x7fa2c8070430 2024-01-21T08:04:30.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.983+0000 7fa2af7fe700 1 -- 172.21.15.46:0/3912295299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fa2b800f070 con 0x7fa2c8070430 2024-01-21T08:04:30.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.983+0000 7fa2af7fe700 1 -- 172.21.15.46:0/3912295299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa2b8025b30 con 0x7fa2c8070430 2024-01-21T08:04:30.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.983+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c8045ad0 con 0x7fa2c8070430 2024-01-21T08:04:30.987 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:30.985+0000 7fa2af7fe700 1 -- 172.21.15.46:0/3912295299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa2b8016070 con 0x7fa2c8070430 2024-01-21T08:04:30.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:29.657305+0000 mon.smithi046 (mon.0) 6 : audit [INF] from='client.? 172.21.15.46:0/1143268896' entity='client.admin' 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.056+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa2c80008d0 con 0x7fa2c8070430 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.056+0000 7fa2af7fe700 1 -- 172.21.15.46:0/3912295299 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fa2b801bdc0 con 0x7fa2c8070430 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsid": "a17ce476-b833-11ee-95b0-87774f69a715", 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "health": { 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T08:04:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 0 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "smithi046" 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T08:04:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "available": false, 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "iostat", 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "restful" 2024-01-21T08:04:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modified": "2024-01-21T08:04:26.123143+0000", 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.057+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 msgr2=0x7fa2c80fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.057+0000 7fa2cd425700 1 --2- 172.21.15.46:0/3912295299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa2b800e040 tx=0x7fa2b8006ee0).stop 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.057+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 shutdown_connections 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.057+0000 7fa2cd425700 1 --2- 172.21.15.46:0/3912295299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2c8070430 0x7fa2c80fc360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.057+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 >> 172.21.15.46:0/3912295299 conn(0x7fa2c80f63c0 msgr2=0x7fa2c80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.057+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 shutdown_connections 2024-01-21T08:04:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:31.057+0000 7fa2cd425700 1 -- 172.21.15.46:0/3912295299 wait complete. 2024-01-21T08:04:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:mgr not available, waiting (1/15)... 2024-01-21T08:04:31.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:31.057273+0000 mon.smithi046 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.46:0/3912295299' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T08:04:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T08:04:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.777+0000 7fded72b3700 1 Processor -- start 2024-01-21T08:04:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.777+0000 7fded72b3700 1 -- start start 2024-01-21T08:04:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fded72b3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fded72b3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded00fee30 con 0x7fded00703d0 2024-01-21T08:04:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fded504f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fded504f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50082/0 (socket says 172.21.15.46:50082) 2024-01-21T08:04:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fded504f700 1 -- 172.21.15.46:0/55356309 learned_addr learned my addr 172.21.15.46:0/55356309 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fded504f700 1 -- 172.21.15.46:0/55356309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded00fef70 con 0x7fded00703d0 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fded504f700 1 --2- 172.21.15.46:0/55356309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fe940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdecc00b5f0 tx=0x7fdecc006ee0).ready entity=mon.0 client_cookie=35e6660e7ad3a647 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.778+0000 7fdebffff700 1 -- 172.21.15.46:0/55356309 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdecc00d600 con 0x7fded00703d0 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fdebffff700 1 -- 172.21.15.46:0/55356309 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdecc00dbd0 con 0x7fded00703d0 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fdebffff700 1 -- 172.21.15.46:0/55356309 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdecc011c00 con 0x7fded00703d0 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fded72b3700 1 -- 172.21.15.46:0/55356309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 msgr2=0x7fded00fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fded72b3700 1 --2- 172.21.15.46:0/55356309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fe940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdecc00b5f0 tx=0x7fdecc006ee0).stop 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fded72b3700 1 -- 172.21.15.46:0/55356309 shutdown_connections 2024-01-21T08:04:33.784 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fded72b3700 1 --2- 172.21.15.46:0/55356309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fe940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fded72b3700 1 -- 172.21.15.46:0/55356309 >> 172.21.15.46:0/55356309 conn(0x7fded00f63c0 msgr2=0x7fded00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fded72b3700 1 -- 172.21.15.46:0/55356309 shutdown_connections 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.779+0000 7fded72b3700 1 -- 172.21.15.46:0/55356309 wait complete. 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded72b3700 1 Processor -- start 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded72b3700 1 -- start start 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded72b3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded72b3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded00fa840 con 0x7fded00703d0 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded504f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded504f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50086/0 (socket says 172.21.15.46:50086) 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded504f700 1 -- 172.21.15.46:0/2451179931 learned_addr learned my addr 172.21.15.46:0/2451179931 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.780+0000 7fded504f700 1 -- 172.21.15.46:0/2451179931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc00b200 con 0x7fded00703d0 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fded504f700 1 --2- 172.21.15.46:0/2451179931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fc3c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdecc00b190 tx=0x7fdecc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fdebdffb700 1 -- 172.21.15.46:0/2451179931 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdecc01b410 con 0x7fded00703d0 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded00fa9e0 con 0x7fded00703d0 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fdebdffb700 1 -- 172.21.15.46:0/2451179931 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdecc01b9e0 con 0x7fded00703d0 2024-01-21T08:04:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fdebdffb700 1 -- 172.21.15.46:0/2451179931 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdecc01a880 con 0x7fded00703d0 2024-01-21T08:04:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded00fadf0 con 0x7fded00703d0 2024-01-21T08:04:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fdebdffb700 1 -- 172.21.15.46:0/2451179931 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fdecc00f070 con 0x7fded00703d0 2024-01-21T08:04:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.781+0000 7fdebdffb700 1 -- 172.21.15.46:0/2451179931 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fdecc025c90 con 0x7fded00703d0 2024-01-21T08:04:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.782+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded0045ad0 con 0x7fded00703d0 2024-01-21T08:04:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.784+0000 7fdebdffb700 1 -- 172.21.15.46:0/2451179931 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fdecc016070 con 0x7fded00703d0 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.855+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fded0104820 con 0x7fded00703d0 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.855+0000 7fdebdffb700 1 -- 172.21.15.46:0/2451179931 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fdecc01bb50 con 0x7fded00703d0 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsid": "a17ce476-b833-11ee-95b0-87774f69a715", 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "health": { 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T08:04:33.865 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 0 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "smithi046" 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:33.866 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T08:04:33.867 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "available": false, 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "iostat", 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "restful" 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:33.868 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modified": "2024-01-21T08:04:26.123143+0000", 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.856+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 msgr2=0x7fded00fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.856+0000 7fded72b3700 1 --2- 172.21.15.46:0/2451179931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fc3c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdecc00b190 tx=0x7fdecc006ee0).stop 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.856+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 shutdown_connections 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.856+0000 7fded72b3700 1 --2- 172.21.15.46:0/2451179931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fded00703d0 0x7fded00fc3c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.856+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 >> 172.21.15.46:0/2451179931 conn(0x7fded00f63c0 msgr2=0x7fded00f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.856+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 shutdown_connections 2024-01-21T08:04:33.869 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:33.856+0000 7fded72b3700 1 -- 172.21.15.46:0/2451179931 wait complete. 2024-01-21T08:04:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:mgr not available, waiting (2/15)... 2024-01-21T08:04:34.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:33.855960+0000 mon.smithi046 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.46:0/2451179931' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T08:04:36.200 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T08:04:36.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:35.733660+0000 mon.smithi046 (mon.0) 9 : cluster [INF] Activating manager daemon smithi046.lmxnwc 2024-01-21T08:04:36.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:35.736255+0000 mon.smithi046 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi046.lmxnwc(active, starting, since 0.00271029s) 2024-01-21T08:04:36.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.736937+0000 mon.smithi046 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T08:04:36.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.737078+0000 mon.smithi046 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T08:04:36.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.737270+0000 mon.smithi046 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T08:04:36.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.737466+0000 mon.smithi046 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-01-21T08:04:36.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.737776+0000 mon.smithi046 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr metadata", "who": "smithi046.lmxnwc", "id": "smithi046.lmxnwc"}]: dispatch 2024-01-21T08:04:36.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:35.751289+0000 mon.smithi046 (mon.0) 16 : cluster [INF] Manager daemon smithi046.lmxnwc is now available 2024-01-21T08:04:36.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.758350+0000 mon.smithi046 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:36.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.763579+0000 mon.smithi046 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:04:36.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.764745+0000 mon.smithi046 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:04:36.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.769927+0000 mon.smithi046 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:36.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:35.771770+0000 mon.smithi046 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.46:0/2815521286' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.522+0000 7fd59f273700 1 Processor -- start 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.522+0000 7fd59f273700 1 -- start start 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.522+0000 7fd59f273700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd5980fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.522+0000 7fd59f273700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980fc5b0 con 0x7fd5980fbca0 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd59d00f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd5980fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd59d00f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd5980fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50134/0 (socket says 172.21.15.46:50134) 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd59d00f700 1 -- 172.21.15.46:0/692641291 learned_addr learned my addr 172.21.15.46:0/692641291 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd59d00f700 1 -- 172.21.15.46:0/692641291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980fc6f0 con 0x7fd5980fbca0 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd59d00f700 1 --2- 172.21.15.46:0/692641291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd5980fc0c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd58800b5f0 tx=0x7fd588006ee0).ready entity=mon.0 client_cookie=b660310b70e9d816 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd587fff700 1 -- 172.21.15.46:0/692641291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58800d600 con 0x7fd5980fbca0 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd587fff700 1 -- 172.21.15.46:0/692641291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd58800dbd0 con 0x7fd5980fbca0 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.523+0000 7fd587fff700 1 -- 172.21.15.46:0/692641291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd588011c00 con 0x7fd5980fbca0 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 -- 172.21.15.46:0/692641291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 msgr2=0x7fd5980fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 --2- 172.21.15.46:0/692641291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd5980fc0c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd58800b5f0 tx=0x7fd588006ee0).stop 2024-01-21T08:04:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 -- 172.21.15.46:0/692641291 shutdown_connections 2024-01-21T08:04:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 --2- 172.21.15.46:0/692641291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd5980fc0c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 -- 172.21.15.46:0/692641291 >> 172.21.15.46:0/692641291 conn(0x7fd5980f74c0 msgr2=0x7fd5980f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 -- 172.21.15.46:0/692641291 shutdown_connections 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 -- 172.21.15.46:0/692641291 wait complete. 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 Processor -- start 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.524+0000 7fd59f273700 1 -- start start 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59f273700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd598072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59f273700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980709f0 con 0x7fd5980fbca0 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59d00f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd598072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59d00f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd598072570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50142/0 (socket says 172.21.15.46:50142) 2024-01-21T08:04:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59d00f700 1 -- 172.21.15.46:0/1111449357 learned_addr learned my addr 172.21.15.46:0/1111449357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59d00f700 1 -- 172.21.15.46:0/1111449357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58800b200 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59d00f700 1 --2- 172.21.15.46:0/1111449357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd598072570 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd58800b1d0 tx=0x7fd588006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd585ffb700 1 -- 172.21.15.46:0/1111449357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58801b410 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd585ffb700 1 -- 172.21.15.46:0/1111449357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd58801b9e0 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd598070b90 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.525+0000 7fd585ffb700 1 -- 172.21.15.46:0/1111449357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58801a840 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.526+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd598070fa0 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.526+0000 7fd585ffb700 1 -- 172.21.15.46:0/1111449357 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fd58800f070 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.526+0000 7fd585ffb700 1 -- 172.21.15.46:0/1111449357 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd588040fe0 con 0x7fd5980fbca0 2024-01-21T08:04:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.527+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd598059100 con 0x7fd5980fbca0 2024-01-21T08:04:36.530 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.529+0000 7fd585ffb700 1 -- 172.21.15.46:0/1111449357 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd588016070 con 0x7fd5980fbca0 2024-01-21T08:04:36.605 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.598+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd598045ad0 con 0x7fd5980fbca0 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.598+0000 7fd585ffb700 1 -- 172.21.15.46:0/1111449357 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fd588022350 con 0x7fd5980fbca0 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsid": "a17ce476-b833-11ee-95b0-87774f69a715", 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "health": { 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 0 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "smithi046" 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T08:04:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T08:04:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "available": false, 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "iostat", 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "restful" 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modified": "2024-01-21T08:04:26.123143+0000", 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.599+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 msgr2=0x7fd598072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.599+0000 7fd59f273700 1 --2- 172.21.15.46:0/1111449357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd598072570 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd58800b1d0 tx=0x7fd588006ee0).stop 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.599+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 shutdown_connections 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.599+0000 7fd59f273700 1 --2- 172.21.15.46:0/1111449357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980fbca0 0x7fd598072570 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.599+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 >> 172.21.15.46:0/1111449357 conn(0x7fd5980f74c0 msgr2=0x7fd5980f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.599+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 shutdown_connections 2024-01-21T08:04:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:36.599+0000 7fd59f273700 1 -- 172.21.15.46:0/1111449357 wait complete. 2024-01-21T08:04:36.929 INFO:teuthology.orchestra.run.smithi046.stderr:mgr not available, waiting (3/15)... 2024-01-21T08:04:36.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:36.598979+0000 mon.smithi046 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.46:0/1111449357' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T08:04:36.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:36.739068+0000 mon.smithi046 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi046.lmxnwc(active, since 1.00553s) 2024-01-21T08:04:38.932 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T08:04:39.062 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:37.788432+0000 mon.smithi046 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi046.lmxnwc(active, since 2s) 2024-01-21T08:04:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.322+0000 7f975f89f700 1 Processor -- start 2024-01-21T08:04:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.322+0000 7f975f89f700 1 -- start start 2024-01-21T08:04:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.322+0000 7f975f89f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f9758100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.322+0000 7f975f89f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97580fc770 con 0x7f9758070390 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.322+0000 7f975d63b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f9758100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.323+0000 7f975d63b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f9758100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50156/0 (socket says 172.21.15.46:50156) 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.323+0000 7f975d63b700 1 -- 172.21.15.46:0/3812150538 learned_addr learned my addr 172.21.15.46:0/3812150538 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.323+0000 7f975d63b700 1 -- 172.21.15.46:0/3812150538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9758101270 con 0x7f9758070390 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.323+0000 7f975d63b700 1 --2- 172.21.15.46:0/3812150538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f9758100d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f974800b5f0 tx=0x7f9748006ee0).ready entity=mon.0 client_cookie=59204e464b6615e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.323+0000 7f9747fff700 1 -- 172.21.15.46:0/3812150538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f974800d600 con 0x7f9758070390 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.323+0000 7f9747fff700 1 -- 172.21.15.46:0/3812150538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f974800dbd0 con 0x7f9758070390 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.323+0000 7f9747fff700 1 -- 172.21.15.46:0/3812150538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9748011c00 con 0x7f9758070390 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.324+0000 7f975f89f700 1 -- 172.21.15.46:0/3812150538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 msgr2=0x7f9758100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.324+0000 7f975f89f700 1 --2- 172.21.15.46:0/3812150538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f9758100d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f974800b5f0 tx=0x7f9748006ee0).stop 2024-01-21T08:04:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.324+0000 7f975f89f700 1 -- 172.21.15.46:0/3812150538 shutdown_connections 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.324+0000 7f975f89f700 1 --2- 172.21.15.46:0/3812150538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f9758100d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.324+0000 7f975f89f700 1 -- 172.21.15.46:0/3812150538 >> 172.21.15.46:0/3812150538 conn(0x7f97580f63c0 msgr2=0x7f97580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.324+0000 7f975f89f700 1 -- 172.21.15.46:0/3812150538 shutdown_connections 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.324+0000 7f975f89f700 1 -- 172.21.15.46:0/3812150538 wait complete. 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975f89f700 1 Processor -- start 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975f89f700 1 -- start start 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975f89f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f975810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975f89f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975810d1b0 con 0x7f9758070390 2024-01-21T08:04:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975d63b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f975810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975d63b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f975810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50160/0 (socket says 172.21.15.46:50160) 2024-01-21T08:04:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975d63b700 1 -- 172.21.15.46:0/927227831 learned_addr learned my addr 172.21.15.46:0/927227831 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975d63b700 1 -- 172.21.15.46:0/927227831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974800b200 con 0x7f9758070390 2024-01-21T08:04:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f975d63b700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f975810ccc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f974800b190 tx=0x7f9748006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.325+0000 7f9745ffb700 1 -- 172.21.15.46:0/927227831 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f974801b410 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.326+0000 7f9745ffb700 1 -- 172.21.15.46:0/927227831 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f974801b9e0 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.326+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975810d3b0 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.326+0000 7f9745ffb700 1 -- 172.21.15.46:0/927227831 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f974801a880 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.326+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975810d7c0 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.327+0000 7f9745ffb700 1 -- 172.21.15.46:0/927227831 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f974800f070 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.327+0000 7f9745ffb700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f974002fd50 0x7f9740032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.327+0000 7f9745ffb700 1 -- 172.21.15.46:0/927227831 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f97480410e0 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.327+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9758045ad0 con 0x7f9758070390 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.327+0000 7f975ce3a700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f974002fd50 0x7f9740032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.327+0000 7f975ce3a700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f974002fd50 0x7f9740032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f97540096a0 tx=0x7f9754006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.332+0000 7f9745ffb700 1 -- 172.21.15.46:0/927227831 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9748016070 con 0x7f9758070390 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.547+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9758101d70 con 0x7f9758070390 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.547+0000 7f9745ffb700 1 -- 172.21.15.46:0/927227831 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9748025460 con 0x7f9758070390 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsid": "a17ce476-b833-11ee-95b0-87774f69a715", 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "health": { 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T08:04:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T08:04:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 0 2024-01-21T08:04:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T08:04:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "smithi046" 2024-01-21T08:04:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-01-21T08:04:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T08:04:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "available": true, 2024-01-21T08:04:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "iostat", 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "restful" 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ], 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "modified": "2024-01-21T08:04:26.123143+0000", 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "services": {} 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: }, 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f974002fd50 msgr2=0x7f9740032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f974002fd50 0x7f9740032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f97540096a0 tx=0x7f9754006b40).stop 2024-01-21T08:04:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 msgr2=0x7f975810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f975810ccc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f974800b190 tx=0x7f9748006ee0).stop 2024-01-21T08:04:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 shutdown_connections 2024-01-21T08:04:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f974002fd50 0x7f9740032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 --2- 172.21.15.46:0/927227831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9758070390 0x7f975810ccc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 >> 172.21.15.46:0/927227831 conn(0x7f97580f63c0 msgr2=0x7f97580f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 shutdown_connections 2024-01-21T08:04:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:39.549+0000 7f975f89f700 1 -- 172.21.15.46:0/927227831 wait complete. 2024-01-21T08:04:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:mgr is available 2024-01-21T08:04:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:Enabling cephadm module... 2024-01-21T08:04:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-21T08:04:40.237 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.233+0000 7f1551896700 1 Processor -- start 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.233+0000 7f1551896700 1 -- start start 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.233+0000 7f1551896700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.233+0000 7f1551896700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0fc5c0 con 0x7f154c0fbcb0 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f154affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f154affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50176/0 (socket says 172.21.15.46:50176) 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f154affd700 1 -- 172.21.15.46:0/1504985745 learned_addr learned my addr 172.21.15.46:0/1504985745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f154affd700 1 -- 172.21.15.46:0/1504985745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0fc700 con 0x7f154c0fbcb0 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f154affd700 1 --2- 172.21.15.46:0/1504985745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0fc0d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f154000b5f0 tx=0x7f1540006ee0).ready entity=mon.0 client_cookie=8ae68d76210aa021 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f1549ffb700 1 -- 172.21.15.46:0/1504985745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f154000d600 con 0x7f154c0fbcb0 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f1549ffb700 1 -- 172.21.15.46:0/1504985745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f154000dbd0 con 0x7f154c0fbcb0 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.234+0000 7f1549ffb700 1 -- 172.21.15.46:0/1504985745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1540011c00 con 0x7f154c0fbcb0 2024-01-21T08:04:40.238 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.235+0000 7f1551896700 1 -- 172.21.15.46:0/1504985745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 msgr2=0x7f154c0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.235+0000 7f1551896700 1 --2- 172.21.15.46:0/1504985745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0fc0d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f154000b5f0 tx=0x7f1540006ee0).stop 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.235+0000 7f1551896700 1 -- 172.21.15.46:0/1504985745 shutdown_connections 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.235+0000 7f1551896700 1 --2- 172.21.15.46:0/1504985745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0fc0d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.235+0000 7f1551896700 1 -- 172.21.15.46:0/1504985745 >> 172.21.15.46:0/1504985745 conn(0x7f154c0f74b0 msgr2=0x7f154c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.235+0000 7f1551896700 1 -- 172.21.15.46:0/1504985745 shutdown_connections 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.235+0000 7f1551896700 1 -- 172.21.15.46:0/1504985745 wait complete. 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f1551896700 1 Processor -- start 2024-01-21T08:04:40.239 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f1551896700 1 -- start start 2024-01-21T08:04:40.241 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f1551896700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:40.241 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f1551896700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c070a60 con 0x7f154c0fbcb0 2024-01-21T08:04:40.241 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f154affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:40.241 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f154affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50186/0 (socket says 172.21.15.46:50186) 2024-01-21T08:04:40.241 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f154affd700 1 -- 172.21.15.46:0/2102332179 learned_addr learned my addr 172.21.15.46:0/2102332179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f154affd700 1 -- 172.21.15.46:0/2102332179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154000b200 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f154affd700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0725e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f154000b1d0 tx=0x7f1540006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.236+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f154001b410 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.237+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f154001b9e0 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.237+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c070c00 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.237+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f154001a840 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.237+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c071010 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.237+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f154000f070 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.237+0000 7f15337fe700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f153802fd50 0x7f1538032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.237+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f15400410e0 con 0x7f154c0fbcb0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.238+0000 7f154a7fc700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f153802fd50 0x7f1538032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.238+0000 7f154a7fc700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f153802fd50 0x7f1538032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f153c0096a0 tx=0x7f153c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:40.242 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.238+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c059100 con 0x7f154c0fbcb0 2024-01-21T08:04:40.244 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.242+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1540022d70 con 0x7f154c0fbcb0 2024-01-21T08:04:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:40.440+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f154c045ad0 con 0x7f154c0fbcb0 2024-01-21T08:04:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:39.548139+0000 mon.smithi046 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.46:0/927227831' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T08:04:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.247+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f15400257e0 con 0x7f154c0fbcb0 2024-01-21T08:04:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.249+0000 7f15337fe700 1 -- 172.21.15.46:0/2102332179 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f15400107d0 con 0x7f154c0fbcb0 2024-01-21T08:04:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f153802fd50 msgr2=0x7f1538032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f153802fd50 0x7f1538032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f153c0096a0 tx=0x7f153c006b40).stop 2024-01-21T08:04:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 msgr2=0x7f154c0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0725e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f154000b1d0 tx=0x7f1540006ee0).stop 2024-01-21T08:04:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 shutdown_connections 2024-01-21T08:04:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f153802fd50 0x7f1538032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 --2- 172.21.15.46:0/2102332179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154c0fbcb0 0x7f154c0725e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.251+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 >> 172.21.15.46:0/2102332179 conn(0x7f154c0f74b0 msgr2=0x7f154c0f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.252+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 shutdown_connections 2024-01-21T08:04:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:41.252+0000 7f1551896700 1 -- 172.21.15.46:0/2102332179 wait complete. 2024-01-21T08:04:41.556 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:40.441639+0000 mon.smithi046 (mon.0) 26 : audit [INF] from='client.? 172.21.15.46:0/2102332179' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-21T08:04:41.564 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.022+0000 7fae60406700 1 Processor -- start 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.022+0000 7fae60406700 1 -- start start 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.022+0000 7fae60406700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae580fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.022+0000 7fae60406700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae580fedd0 con 0x7fae58070430 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.022+0000 7fae5e1a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae580fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.022+0000 7fae5e1a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae580fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54474/0 (socket says 172.21.15.46:54474) 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.022+0000 7fae5e1a2700 1 -- 172.21.15.46:0/1083888412 learned_addr learned my addr 172.21.15.46:0/1083888412 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.023+0000 7fae5e1a2700 1 -- 172.21.15.46:0/1083888412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580fef10 con 0x7fae58070430 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.023+0000 7fae5e1a2700 1 --2- 172.21.15.46:0/1083888412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae580fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fae5400b5f0 tx=0x7fae54006ee0).ready entity=mon.0 client_cookie=8faa0c690b83b042 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.023+0000 7fae5d1a0700 1 -- 172.21.15.46:0/1083888412 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae5400d600 con 0x7fae58070430 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.023+0000 7fae5d1a0700 1 -- 172.21.15.46:0/1083888412 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fae5400dbd0 con 0x7fae58070430 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.023+0000 7fae5d1a0700 1 -- 172.21.15.46:0/1083888412 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae54011c00 con 0x7fae58070430 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 -- 172.21.15.46:0/1083888412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 msgr2=0x7fae580fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 --2- 172.21.15.46:0/1083888412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae580fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fae5400b5f0 tx=0x7fae54006ee0).stop 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 -- 172.21.15.46:0/1083888412 shutdown_connections 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 --2- 172.21.15.46:0/1083888412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae580fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 -- 172.21.15.46:0/1083888412 >> 172.21.15.46:0/1083888412 conn(0x7fae580f63c0 msgr2=0x7fae580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 -- 172.21.15.46:0/1083888412 shutdown_connections 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 -- 172.21.15.46:0/1083888412 wait complete. 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.024+0000 7fae60406700 1 Processor -- start 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae60406700 1 -- start start 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae60406700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae5810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae60406700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5810aef0 con 0x7fae58070430 2024-01-21T08:04:42.120 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae5e1a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae5810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae5e1a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae5810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54484/0 (socket says 172.21.15.46:54484) 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae5e1a2700 1 -- 172.21.15.46:0/3272281194 learned_addr learned my addr 172.21.15.46:0/3272281194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae5e1a2700 1 -- 172.21.15.46:0/3272281194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5400b200 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae5e1a2700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae5810aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fae5400b190 tx=0x7fae54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae46ffd700 1 -- 172.21.15.46:0/3272281194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae5401b410 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae46ffd700 1 -- 172.21.15.46:0/3272281194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fae5401b9e0 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5810b0f0 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae46ffd700 1 -- 172.21.15.46:0/3272281194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae5401a880 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.025+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5810b500 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.026+0000 7fae46ffd700 1 -- 172.21.15.46:0/3272281194 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fae5400f070 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.026+0000 7fae46ffd700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 0x7fae40032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.026+0000 7fae46ffd700 1 -- 172.21.15.46:0/3272281194 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fae540420a0 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.026+0000 7fae5d9a1700 1 -- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 msgr2=0x7fae40032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1991662361 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.026+0000 7fae5d9a1700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 0x7fae40032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.027+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae58059100 con 0x7fae58070430 2024-01-21T08:04:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.031+0000 7fae46ffd700 1 -- 172.21.15.46:0/3272281194 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae54010360 con 0x7fae58070430 2024-01-21T08:04:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.227+0000 7fae5d9a1700 1 -- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 msgr2=0x7fae40032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1991662361 2024-01-21T08:04:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.227+0000 7fae5d9a1700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 0x7fae40032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T08:04:42.231 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.229+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fae581047f0 con 0x7fae58070430 2024-01-21T08:04:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.230+0000 7fae46ffd700 1 -- 172.21.15.46:0/3272281194 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fae54022b00 con 0x7fae58070430 2024-01-21T08:04:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:04:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-21T08:04:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "available": true, 2024-01-21T08:04:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "active_name": "smithi046.lmxnwc", 2024-01-21T08:04:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T08:04:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:04:42.234 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 msgr2=0x7fae40032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T08:04:42.234 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 0x7fae40032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:04:42.234 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 msgr2=0x7fae5810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:42.234 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae5810aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fae5400b190 tx=0x7fae54006ee0).stop 2024-01-21T08:04:42.234 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 shutdown_connections 2024-01-21T08:04:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7fae4002fd50 0x7fae40032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:04:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 --2- 172.21.15.46:0/3272281194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae58070430 0x7fae5810aa00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 >> 172.21.15.46:0/3272281194 conn(0x7fae580f63c0 msgr2=0x7fae580f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 shutdown_connections 2024-01-21T08:04:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:42.232+0000 7fae60406700 1 -- 172.21.15.46:0/3272281194 wait complete. 2024-01-21T08:04:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:41.249909+0000 mon.smithi046 (mon.0) 27 : audit [INF] from='client.? 172.21.15.46:0/2102332179' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-21T08:04:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:41.249993+0000 mon.smithi046 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi046.lmxnwc(active, since 5s) 2024-01-21T08:04:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:42.230808+0000 mon.smithi046 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.46:0/3272281194' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T08:04:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for the mgr to restart... 2024-01-21T08:04:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for mgr epoch 5... 2024-01-21T08:04:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T08:04:44.407 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.403+0000 7f04e528c700 1 Processor -- start 2024-01-21T08:04:44.407 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.403+0000 7f04e528c700 1 -- start start 2024-01-21T08:04:44.407 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.403+0000 7f04e528c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:44.407 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.403+0000 7f04e528c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00fedd0 con 0x7f04e0070430 2024-01-21T08:04:44.407 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.403+0000 7f04deffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:44.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.403+0000 7f04deffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54494/0 (socket says 172.21.15.46:54494) 2024-01-21T08:04:44.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.403+0000 7f04deffd700 1 -- 172.21.15.46:0/1892751613 learned_addr learned my addr 172.21.15.46:0/1892751613 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:44.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.404+0000 7f04deffd700 1 -- 172.21.15.46:0/1892751613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00fef10 con 0x7f04e0070430 2024-01-21T08:04:44.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.404+0000 7f04deffd700 1 --2- 172.21.15.46:0/1892751613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fe8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f04d000b5f0 tx=0x7f04d0006ee0).ready entity=mon.0 client_cookie=1eef1c1dcc81dc0e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:44.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.404+0000 7f04ddffb700 1 -- 172.21.15.46:0/1892751613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d000d600 con 0x7f04e0070430 2024-01-21T08:04:44.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.404+0000 7f04ddffb700 1 -- 172.21.15.46:0/1892751613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f04d000dbd0 con 0x7f04e0070430 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.404+0000 7f04ddffb700 1 -- 172.21.15.46:0/1892751613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d0011c00 con 0x7f04e0070430 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 -- 172.21.15.46:0/1892751613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 msgr2=0x7f04e00fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 --2- 172.21.15.46:0/1892751613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fe8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f04d000b5f0 tx=0x7f04d0006ee0).stop 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 -- 172.21.15.46:0/1892751613 shutdown_connections 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 --2- 172.21.15.46:0/1892751613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fe8e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 -- 172.21.15.46:0/1892751613 >> 172.21.15.46:0/1892751613 conn(0x7f04e00f63c0 msgr2=0x7f04e00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 -- 172.21.15.46:0/1892751613 shutdown_connections 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 -- 172.21.15.46:0/1892751613 wait complete. 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.405+0000 7f04e528c700 1 Processor -- start 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04e528c700 1 -- start start 2024-01-21T08:04:44.412 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04e528c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04e528c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00fa7e0 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04deffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04deffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54506/0 (socket says 172.21.15.46:54506) 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04deffd700 1 -- 172.21.15.46:0/3956527973 learned_addr learned my addr 172.21.15.46:0/3956527973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04deffd700 1 -- 172.21.15.46:0/3956527973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d000b200 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04deffd700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fc360 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f04d000e040 tx=0x7f04d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d000d9f0 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f04d001a9f0 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e00fa980 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.406+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d001b8c0 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.407+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e00fad90 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.407+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f04d000f070 con 0x7f04e0070430 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.408+0000 7f04c77fe700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 0x7f04cc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.408+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 --> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04cc032930 con 0x7f04cc02fdb0 2024-01-21T08:04:44.413 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.408+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f04d00411d0 con 0x7f04e0070430 2024-01-21T08:04:44.414 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.408+0000 7f04de7fc700 1 -- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 msgr2=0x7f04cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1991662361 2024-01-21T08:04:44.414 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.408+0000 7f04de7fc700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 0x7f04cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T08:04:44.612 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.609+0000 7f04de7fc700 1 -- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 msgr2=0x7f04cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1991662361 2024-01-21T08:04:44.612 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:44.609+0000 7f04de7fc700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 0x7f04cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T08:04:45.013 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:45.010+0000 7f04de7fc700 1 -- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 msgr2=0x7f04cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1991662361 2024-01-21T08:04:45.013 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:45.010+0000 7f04de7fc700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 0x7f04cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T08:04:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:45.811+0000 7f04de7fc700 1 -- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 msgr2=0x7f04cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1991662361 2024-01-21T08:04:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:45.811+0000 7f04de7fc700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 0x7f04cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T08:04:46.472 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:46.468+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f04d001ba20 con 0x7f04e0070430 2024-01-21T08:04:46.472 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:46.469+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 msgr2=0x7f04cc032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T08:04:46.472 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:46.469+0000 7f04c77fe700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 0x7f04cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:04:47.473 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.470+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f04d00257e0 con 0x7f04e0070430 2024-01-21T08:04:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.470+0000 7f04c77fe700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f04cc033880 0x7f04cc035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.470+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04cc032930 con 0x7f04cc033880 2024-01-21T08:04:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.471+0000 7f04de7fc700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f04cc033880 0x7f04cc035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.472+0000 7f04de7fc700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f04cc033880 0x7f04cc035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f04d40036d0 tx=0x7f04d4006d20).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.472+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f04cc032930 con 0x7f04cc033880 2024-01-21T08:04:47.481 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f04e0059100 con 0x7f04cc033880 2024-01-21T08:04:47.483 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04c77fe700 1 -- 172.21.15.46:0/3956527973 <== mgr.14116 v2:172.21.15.46:6800/2245961460 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f04e0059100 con 0x7f04cc033880 2024-01-21T08:04:47.483 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:04:47.483 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-21T08:04:47.483 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f04cc033880 msgr2=0x7f04cc035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f04cc033880 0x7f04cc035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f04d40036d0 tx=0x7f04d4006d20).stop 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 msgr2=0x7f04e00fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fc360 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f04d000e040 tx=0x7f04d0006ee0).stop 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 shutdown_connections 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f04cc033880 0x7f04cc035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0070430 0x7f04e00fc360 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 --2- 172.21.15.46:0/3956527973 >> [v2:172.21.15.46:6800/1991662361,v1:172.21.15.46:6801/1991662361] conn(0x7f04cc02fdb0 0x7f04cc032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.479+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 >> 172.21.15.46:0/3956527973 conn(0x7f04e00f63c0 msgr2=0x7f04e00f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.480+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 shutdown_connections 2024-01-21T08:04:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:47.480+0000 7f04e528c700 1 -- 172.21.15.46:0/3956527973 wait complete. 2024-01-21T08:04:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:46.418637+0000 mon.smithi046 (mon.0) 30 : cluster [INF] Active manager daemon smithi046.lmxnwc restarted 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:46.419021+0000 mon.smithi046 (mon.0) 31 : cluster [INF] Activating manager daemon smithi046.lmxnwc 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:46.420657+0000 mon.smithi046 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:46.471507+0000 mon.smithi046 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi046.lmxnwc(active, starting, since 0.0525597s) 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.472344+0000 mon.smithi046 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.472995+0000 mon.smithi046 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr metadata", "who": "smithi046.lmxnwc", "id": "smithi046.lmxnwc"}]: dispatch 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.473694+0000 mon.smithi046 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.473926+0000 mon.smithi046 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.474162+0000 mon.smithi046 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:46.488287+0000 mon.smithi046 (mon.0) 39 : cluster [INF] Manager daemon smithi046.lmxnwc is now available 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.501373+0000 mon.smithi046 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:47.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.504000+0000 mon.smithi046 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:47.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.506382+0000 mon.smithi046 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:47.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.600462+0000 mon.smithi046 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:47.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.601111+0000 mon.smithi046 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:04:47.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.602831+0000 mon.smithi046 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:04:47.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.604272+0000 mon.smithi046 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:04:47.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.614037+0000 mon.smithi046 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:04:47.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:46.615258+0000 mon.smithi046 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:04:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:mgr epoch 5 is available 2024-01-21T08:04:47.918 INFO:teuthology.orchestra.run.smithi046.stderr:Setting orchestrator backend to cephadm... 2024-01-21T08:04:47.918 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.171+0000 7effc421e700 1 Processor -- start 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.171+0000 7effc421e700 1 -- start start 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc421e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc421e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0fb490 con 0x7effbc0fab80 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc1fba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc1fba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54560/0 (socket says 172.21.15.46:54560) 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc1fba700 1 -- 172.21.15.46:0/1317472804 learned_addr learned my addr 172.21.15.46:0/1317472804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc1fba700 1 -- 172.21.15.46:0/1317472804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc0fb5d0 con 0x7effbc0fab80 2024-01-21T08:04:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc1fba700 1 --2- 172.21.15.46:0/1317472804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc0fafa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7effb800b5f0 tx=0x7effb8006ee0).ready entity=mon.0 client_cookie=b64465d3afe3b36f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc0fb8700 1 -- 172.21.15.46:0/1317472804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb800d600 con 0x7effbc0fab80 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc0fb8700 1 -- 172.21.15.46:0/1317472804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7effb800dbd0 con 0x7effbc0fab80 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.172+0000 7effc0fb8700 1 -- 172.21.15.46:0/1317472804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb8011c00 con 0x7effbc0fab80 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.173+0000 7effc421e700 1 -- 172.21.15.46:0/1317472804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 msgr2=0x7effbc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.173+0000 7effc421e700 1 --2- 172.21.15.46:0/1317472804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc0fafa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7effb800b5f0 tx=0x7effb8006ee0).stop 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.173+0000 7effc421e700 1 -- 172.21.15.46:0/1317472804 shutdown_connections 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.173+0000 7effc421e700 1 --2- 172.21.15.46:0/1317472804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc0fafa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.173+0000 7effc421e700 1 -- 172.21.15.46:0/1317472804 >> 172.21.15.46:0/1317472804 conn(0x7effbc0f63c0 msgr2=0x7effbc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.173+0000 7effc421e700 1 -- 172.21.15.46:0/1317472804 shutdown_connections 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.173+0000 7effc421e700 1 -- 172.21.15.46:0/1317472804 wait complete. 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.174+0000 7effc421e700 1 Processor -- start 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.174+0000 7effc421e700 1 -- start start 2024-01-21T08:04:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.174+0000 7effc421e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc10d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.174+0000 7effc421e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc070470 con 0x7effbc0fab80 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.174+0000 7effc1fba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc10d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.174+0000 7effc1fba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc10d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54562/0 (socket says 172.21.15.46:54562) 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.174+0000 7effc1fba700 1 -- 172.21.15.46:0/2085505887 learned_addr learned my addr 172.21.15.46:0/2085505887 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.175+0000 7effc1fba700 1 -- 172.21.15.46:0/2085505887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb800b200 con 0x7effbc0fab80 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.175+0000 7effc1fba700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc10d6e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7effb800b1d0 tx=0x7effb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.175+0000 7effae7fc700 1 -- 172.21.15.46:0/2085505887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb801b410 con 0x7effbc0fab80 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.175+0000 7effae7fc700 1 -- 172.21.15.46:0/2085505887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7effb801b9e0 con 0x7effbc0fab80 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.175+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc070670 con 0x7effbc0fab80 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.175+0000 7effae7fc700 1 -- 172.21.15.46:0/2085505887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb801a840 con 0x7effbc0fab80 2024-01-21T08:04:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.175+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc10dd90 con 0x7effbc0fab80 2024-01-21T08:04:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.176+0000 7effae7fc700 1 -- 172.21.15.46:0/2085505887 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7effb800f070 con 0x7effbc0fab80 2024-01-21T08:04:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.176+0000 7effae7fc700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7effa402fca0 0x7effa4032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.176+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc045ad0 con 0x7effbc0fab80 2024-01-21T08:04:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.176+0000 7effae7fc700 1 -- 172.21.15.46:0/2085505887 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7effb80411f0 con 0x7effbc0fab80 2024-01-21T08:04:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.176+0000 7effc17b9700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7effa402fca0 0x7effa4032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.177+0000 7effc17b9700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7effa402fca0 0x7effa4032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7effb00096a0 tx=0x7effb0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:48.183 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.181+0000 7effae7fc700 1 -- 172.21.15.46:0/2085505887 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effb8019af0 con 0x7effbc0fab80 2024-01-21T08:04:48.374 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.371+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7effbc057f10 con 0x7effa402fca0 2024-01-21T08:04:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.376+0000 7effae7fc700 1 -- 172.21.15.46:0/2085505887 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7effbc057f10 con 0x7effa402fca0 2024-01-21T08:04:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7effa402fca0 msgr2=0x7effa4032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7effa402fca0 0x7effa4032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7effb00096a0 tx=0x7effb0006b40).stop 2024-01-21T08:04:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 msgr2=0x7effbc10d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc10d6e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7effb800b1d0 tx=0x7effb8006ee0).stop 2024-01-21T08:04:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 shutdown_connections 2024-01-21T08:04:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7effa402fca0 0x7effa4032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 --2- 172.21.15.46:0/2085505887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effbc0fab80 0x7effbc10d6e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 >> 172.21.15.46:0/2085505887 conn(0x7effbc0f63c0 msgr2=0x7effbc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 shutdown_connections 2024-01-21T08:04:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:48.378+0000 7effc421e700 1 -- 172.21.15.46:0/2085505887 wait complete. 2024-01-21T08:04:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-21T08:04:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:47.473954+0000 mgr.smithi046.lmxnwc (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T08:04:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:47.474531+0000 mon.smithi046 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi046.lmxnwc(active, since 1.05558s) 2024-01-21T08:04:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:47.480855+0000 mgr.smithi046.lmxnwc (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T08:04:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:48.377235+0000 mon.smithi046 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:48.380353+0000 mon.smithi046 (mon.0) 51 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:04:49.060 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.054+0000 7feb48a56700 1 Processor -- start 2024-01-21T08:04:49.060 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.054+0000 7feb48a56700 1 -- start start 2024-01-21T08:04:49.060 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb48a56700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:49.060 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb48a56700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440fede0 con 0x7feb440704a0 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54564/0 (socket says 172.21.15.46:54564) 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4259c700 1 -- 172.21.15.46:0/2182733731 learned_addr learned my addr 172.21.15.46:0/2182733731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4259c700 1 -- 172.21.15.46:0/2182733731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb440fef20 con 0x7feb440704a0 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4259c700 1 --2- 172.21.15.46:0/2182733731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fe8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feb3400b5f0 tx=0x7feb34006ee0).ready entity=mon.0 client_cookie=e9001aa6e7aeea69 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4159a700 1 -- 172.21.15.46:0/2182733731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb3400d600 con 0x7feb440704a0 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4159a700 1 -- 172.21.15.46:0/2182733731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb3400dbd0 con 0x7feb440704a0 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.055+0000 7feb4159a700 1 -- 172.21.15.46:0/2182733731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb34011c00 con 0x7feb440704a0 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.056+0000 7feb48a56700 1 -- 172.21.15.46:0/2182733731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 msgr2=0x7feb440fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.056+0000 7feb48a56700 1 --2- 172.21.15.46:0/2182733731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fe8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feb3400b5f0 tx=0x7feb34006ee0).stop 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.056+0000 7feb48a56700 1 -- 172.21.15.46:0/2182733731 shutdown_connections 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.056+0000 7feb48a56700 1 --2- 172.21.15.46:0/2182733731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fe8f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.056+0000 7feb48a56700 1 -- 172.21.15.46:0/2182733731 >> 172.21.15.46:0/2182733731 conn(0x7feb440f63b0 msgr2=0x7feb440f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.056+0000 7feb48a56700 1 -- 172.21.15.46:0/2182733731 shutdown_connections 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.056+0000 7feb48a56700 1 -- 172.21.15.46:0/2182733731 wait complete. 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.057+0000 7feb48a56700 1 Processor -- start 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.057+0000 7feb48a56700 1 -- start start 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.057+0000 7feb48a56700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.057+0000 7feb48a56700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440fa7f0 con 0x7feb440704a0 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.057+0000 7feb4259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.057+0000 7feb4259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54574/0 (socket says 172.21.15.46:54574) 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.057+0000 7feb4259c700 1 -- 172.21.15.46:0/3793666114 learned_addr learned my addr 172.21.15.46:0/3793666114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.058+0000 7feb4259c700 1 -- 172.21.15.46:0/3793666114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3400b200 con 0x7feb440704a0 2024-01-21T08:04:49.062 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.058+0000 7feb4259c700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fc370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7feb3400e040 tx=0x7feb34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.058+0000 7feb3affd700 1 -- 172.21.15.46:0/3793666114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb3400d9f0 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.058+0000 7feb3affd700 1 -- 172.21.15.46:0/3793666114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb3401a9f0 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.058+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb440fa990 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.058+0000 7feb3affd700 1 -- 172.21.15.46:0/3793666114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb3401b8c0 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.058+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb440fada0 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.059+0000 7feb3affd700 1 -- 172.21.15.46:0/3793666114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7feb3400f070 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.059+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb44045ad0 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.059+0000 7feb3affd700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb2402fd10 0x7feb240321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.059+0000 7feb3affd700 1 -- 172.21.15.46:0/3793666114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7feb340411d0 con 0x7feb440704a0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.059+0000 7feb41d9b700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb2402fd10 0x7feb240321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.060+0000 7feb41d9b700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb2402fd10 0x7feb240321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7feb2c0096a0 tx=0x7feb2c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.063+0000 7feb3affd700 1 -- 172.21.15.46:0/3793666114 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb34006a70 con 0x7feb440704a0 2024-01-21T08:04:49.224 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.221+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7feb440fafe0 con 0x7feb2402fd10 2024-01-21T08:04:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.222+0000 7feb3affd700 1 -- 172.21.15.46:0/3793666114 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7feb440fafe0 con 0x7feb2402fd10 2024-01-21T08:04:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: value unchanged 2024-01-21T08:04:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.223+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb2402fd10 msgr2=0x7feb240321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.223+0000 7feb48a56700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb2402fd10 0x7feb240321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7feb2c0096a0 tx=0x7feb2c006b40).stop 2024-01-21T08:04:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.223+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 msgr2=0x7feb440fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.223+0000 7feb48a56700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fc370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7feb3400e040 tx=0x7feb34006ee0).stop 2024-01-21T08:04:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.223+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 shutdown_connections 2024-01-21T08:04:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.224+0000 7feb48a56700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb2402fd10 0x7feb240321d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.224+0000 7feb48a56700 1 --2- 172.21.15.46:0/3793666114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb440704a0 0x7feb440fc370 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:49.228 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.224+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 >> 172.21.15.46:0/3793666114 conn(0x7feb440f63b0 msgr2=0x7feb440f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:49.228 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.224+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 shutdown_connections 2024-01-21T08:04:49.228 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:49.224+0000 7feb48a56700 1 -- 172.21.15.46:0/3793666114 wait complete. 2024-01-21T08:04:49.866 INFO:teuthology.orchestra.run.smithi046.stderr:Generating ssh key... 2024-01-21T08:04:49.866 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-21T08:04:50.613 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:48.373474+0000 mgr.smithi046.lmxnwc (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:04:50.613 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:49.223771+0000 mgr.smithi046.lmxnwc (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:04:50.614 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:04:49.378040+0000 mon.smithi046 (mon.0) 52 : cluster [DBG] mgrmap e8: smithi046.lmxnwc(active, since 2s) 2024-01-21T08:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.237+0000 7f0209089700 1 Processor -- start 2024-01-21T08:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.237+0000 7f0209089700 1 -- start start 2024-01-21T08:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.237+0000 7f0209089700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f02040fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.237+0000 7f0209089700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02040fedd0 con 0x7f0204070430 2024-01-21T08:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0202d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f02040fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0202d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f02040fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50734/0 (socket says 172.21.15.46:50734) 2024-01-21T08:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0202d9d700 1 -- 172.21.15.46:0/1800021117 learned_addr learned my addr 172.21.15.46:0/1800021117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0202d9d700 1 -- 172.21.15.46:0/1800021117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02040fef10 con 0x7f0204070430 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0202d9d700 1 --2- 172.21.15.46:0/1800021117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f02040fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f01f400b5f0 tx=0x7f01f4006ee0).ready entity=mon.0 client_cookie=322a0167fedd56d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0201d9b700 1 -- 172.21.15.46:0/1800021117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01f400d600 con 0x7f0204070430 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0201d9b700 1 -- 172.21.15.46:0/1800021117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f01f400dbd0 con 0x7f0204070430 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.238+0000 7f0201d9b700 1 -- 172.21.15.46:0/1800021117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01f4011c00 con 0x7f0204070430 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.239+0000 7f0209089700 1 -- 172.21.15.46:0/1800021117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 msgr2=0x7f02040fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.239+0000 7f0209089700 1 --2- 172.21.15.46:0/1800021117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f02040fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f01f400b5f0 tx=0x7f01f4006ee0).stop 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.239+0000 7f0209089700 1 -- 172.21.15.46:0/1800021117 shutdown_connections 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.239+0000 7f0209089700 1 --2- 172.21.15.46:0/1800021117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f02040fe8e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.239+0000 7f0209089700 1 -- 172.21.15.46:0/1800021117 >> 172.21.15.46:0/1800021117 conn(0x7f02040f63c0 msgr2=0x7f02040f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.239+0000 7f0209089700 1 -- 172.21.15.46:0/1800021117 shutdown_connections 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.239+0000 7f0209089700 1 -- 172.21.15.46:0/1800021117 wait complete. 2024-01-21T08:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.240+0000 7f0209089700 1 Processor -- start 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.240+0000 7f0209089700 1 -- start start 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.240+0000 7f0209089700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f020410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.240+0000 7f0209089700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020410aef0 con 0x7f0204070430 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.240+0000 7f0202d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f020410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.240+0000 7f0202d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f020410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50750/0 (socket says 172.21.15.46:50750) 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.240+0000 7f0202d9d700 1 -- 172.21.15.46:0/3135293188 learned_addr learned my addr 172.21.15.46:0/3135293188 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.241+0000 7f0202d9d700 1 -- 172.21.15.46:0/3135293188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f400b200 con 0x7f0204070430 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.241+0000 7f0202d9d700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f020410aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f01f400b190 tx=0x7f01f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.241+0000 7f01fb7fe700 1 -- 172.21.15.46:0/3135293188 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01f401b410 con 0x7f0204070430 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.241+0000 7f01fb7fe700 1 -- 172.21.15.46:0/3135293188 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f01f401b9e0 con 0x7f0204070430 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.241+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020410b0f0 con 0x7f0204070430 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.241+0000 7f01fb7fe700 1 -- 172.21.15.46:0/3135293188 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01f401a880 con 0x7f0204070430 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.241+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020410b500 con 0x7f0204070430 2024-01-21T08:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.242+0000 7f01fb7fe700 1 -- 172.21.15.46:0/3135293188 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f01f400f070 con 0x7f0204070430 2024-01-21T08:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.242+0000 7f01fb7fe700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f01e402fdb0 0x7f01e4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.242+0000 7f01fb7fe700 1 -- 172.21.15.46:0/3135293188 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f01f4041200 con 0x7f0204070430 2024-01-21T08:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.242+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0204045ad0 con 0x7f0204070430 2024-01-21T08:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.242+0000 7f020259c700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f01e402fdb0 0x7f01e4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.242+0000 7f020259c700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f01e402fdb0 0x7f01e4032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f01ec0096a0 tx=0x7f01ec006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.246+0000 7f01fb7fe700 1 -- 172.21.15.46:0/3135293188 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01f4016070 con 0x7f0204070430 2024-01-21T08:04:51.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:51.406+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0204104900 con 0x7f01e402fdb0 2024-01-21T08:04:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.167+0000 7f01fb7fe700 1 -- 172.21.15.46:0/3135293188 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0204104900 con 0x7f01e402fdb0 2024-01-21T08:04:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f01e402fdb0 msgr2=0x7f01e4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f01e402fdb0 0x7f01e4032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f01ec0096a0 tx=0x7f01ec006b40).stop 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 msgr2=0x7f020410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f020410aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f01f400b190 tx=0x7f01f4006ee0).stop 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 shutdown_connections 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f01e402fdb0 0x7f01e4032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 --2- 172.21.15.46:0/3135293188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0204070430 0x7f020410aa00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 >> 172.21.15.46:0/3135293188 conn(0x7f02040f63c0 msgr2=0x7f02040f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 shutdown_connections 2024-01-21T08:04:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:52.169+0000 7f0209089700 1 -- 172.21.15.46:0/3135293188 wait complete. 2024-01-21T08:04:53.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:04:51.408252+0000 mgr.smithi046.lmxnwc (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2024-01-21T08:04:53.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:52.166626+0000 mon.smithi046 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:53.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:52.168403+0000 mon.smithi046 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:54.803 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-21T08:04:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.091+0000 7f6512242700 1 Processor -- start 2024-01-21T08:04:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.091+0000 7f6512242700 1 -- start start 2024-01-21T08:04:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f6512242700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f6512242700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c0fb490 con 0x7f650c0fab80 2024-01-21T08:04:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50756/0 (socket says 172.21.15.46:50756) 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650b7fe700 1 -- 172.21.15.46:0/2543748252 learned_addr learned my addr 172.21.15.46:0/2543748252 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650b7fe700 1 -- 172.21.15.46:0/2543748252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c0fb5d0 con 0x7f650c0fab80 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650b7fe700 1 --2- 172.21.15.46:0/2543748252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c0fafa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f64fc00b870 tx=0x7f64fc006f70).ready entity=mon.0 client_cookie=106fc2b770f2820e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650a7fc700 1 -- 172.21.15.46:0/2543748252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64fc00da20 con 0x7f650c0fab80 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650a7fc700 1 -- 172.21.15.46:0/2543748252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f64fc010470 con 0x7f650c0fab80 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.092+0000 7f650a7fc700 1 -- 172.21.15.46:0/2543748252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64fc00f5b0 con 0x7f650c0fab80 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.093+0000 7f6512242700 1 -- 172.21.15.46:0/2543748252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 msgr2=0x7f650c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.093+0000 7f6512242700 1 --2- 172.21.15.46:0/2543748252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c0fafa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f64fc00b870 tx=0x7f64fc006f70).stop 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.093+0000 7f6512242700 1 -- 172.21.15.46:0/2543748252 shutdown_connections 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.093+0000 7f6512242700 1 --2- 172.21.15.46:0/2543748252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c0fafa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.093+0000 7f6512242700 1 -- 172.21.15.46:0/2543748252 >> 172.21.15.46:0/2543748252 conn(0x7f650c0f63c0 msgr2=0x7f650c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.093+0000 7f6512242700 1 -- 172.21.15.46:0/2543748252 shutdown_connections 2024-01-21T08:04:56.213 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.093+0000 7f6512242700 1 -- 172.21.15.46:0/2543748252 wait complete. 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.094+0000 7f6512242700 1 Processor -- start 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.094+0000 7f6512242700 1 -- start start 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.094+0000 7f6512242700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c10cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.094+0000 7f6512242700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c10d090 con 0x7f650c0fab80 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.094+0000 7f650b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c10cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.094+0000 7f650b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c10cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50768/0 (socket says 172.21.15.46:50768) 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.094+0000 7f650b7fe700 1 -- 172.21.15.46:0/1053705801 learned_addr learned my addr 172.21.15.46:0/1053705801 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.095+0000 7f650b7fe700 1 -- 172.21.15.46:0/1053705801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc00b480 con 0x7f650c0fab80 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.095+0000 7f650b7fe700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c10cba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f64fc006a40 tx=0x7f64fc006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.095+0000 7f6503fff700 1 -- 172.21.15.46:0/1053705801 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64fc00de00 con 0x7f650c0fab80 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.095+0000 7f6503fff700 1 -- 172.21.15.46:0/1053705801 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f64fc00fef0 con 0x7f650c0fab80 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.095+0000 7f6503fff700 1 -- 172.21.15.46:0/1053705801 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64fc019e90 con 0x7f650c0fab80 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.095+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c10d290 con 0x7f650c0fab80 2024-01-21T08:04:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.095+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c10d6a0 con 0x7f650c0fab80 2024-01-21T08:04:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.096+0000 7f6503fff700 1 -- 172.21.15.46:0/1053705801 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f64fc02a070 con 0x7f650c0fab80 2024-01-21T08:04:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.096+0000 7f6503fff700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f64ec02fd50 0x7f64ec032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.096+0000 7f6503fff700 1 -- 172.21.15.46:0/1053705801 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f64fc041b10 con 0x7f650c0fab80 2024-01-21T08:04:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.096+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c045ad0 con 0x7f650c0fab80 2024-01-21T08:04:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.096+0000 7f650affd700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f64ec02fd50 0x7f64ec032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.096+0000 7f650affd700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f64ec02fd50 0x7f64ec032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f64f40096a0 tx=0x7f64f4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.100+0000 7f6503fff700 1 -- 172.21.15.46:0/1053705801 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64fc014020 con 0x7f650c0fab80 2024-01-21T08:04:56.263 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.261+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f650c106af0 con 0x7f64ec02fd50 2024-01-21T08:04:56.264 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.261+0000 7f6503fff700 1 -- 172.21.15.46:0/1053705801 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f650c106af0 con 0x7f64ec02fd50 2024-01-21T08:04:56.264 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUZNjkANzwqvxqxiv4P3kOpOtqccmeBY2Nw/61ji5JLjt6Rfv5fM4UcykVC5/8YGE5YzleGduKRZYwo1/J/jQWrsEKOKnWUdnRneP4rB4Yc7fRk8oZrBj6i9hfb7yoKFKKLTDTjfW7qM3bumW0LaB5OVSHbu4EFyvsNPbWGtP6Qs1vmyMD4zXNBcVajv61g8bWecX4odTVQtlAvK6sP/9QG3c1mB7TU2FIQSys58enw6RA+cG6JJC7vTdXxkLSj/tnUYpvFuAwU3b4DHJBWfolog+SE2oh4Ya0T7j6GPVd6w/XYzzzxv90FEhc9unFp06lkfAbATVadlMp4d6IJbWutSfsZt3/5ZHmVKcsM8Lu0dAJiD7mGcXQ8ApAneEjrWJ1YUVvRY922h9PVZZO3NuREtl/8yeWFteG5i+CniRQ1rJxUFLjkt9sFMdjBg1g76+Djd4bnhkpoG9frR+6UpxhXbgtRHd8AIPZq6RIGReuEo6jYtylft1N/Ao4UChgxOU= ceph-a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:04:56.266 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f64ec02fd50 msgr2=0x7f64ec032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:56.266 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f64ec02fd50 0x7f64ec032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f64f40096a0 tx=0x7f64f4006b40).stop 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 msgr2=0x7f650c10cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c10cba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f64fc006a40 tx=0x7f64fc006f70).stop 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 shutdown_connections 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f64ec02fd50 0x7f64ec032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 --2- 172.21.15.46:0/1053705801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f650c0fab80 0x7f650c10cba0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 >> 172.21.15.46:0/1053705801 conn(0x7f650c0f63c0 msgr2=0x7f650c103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 shutdown_connections 2024-01-21T08:04:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:56.263+0000 7f6512242700 1 -- 172.21.15.46:0/1053705801 wait complete. 2024-01-21T08:04:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:56.263283+0000 mgr.smithi046.lmxnwc (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:04:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-21T08:04:57.671 INFO:teuthology.orchestra.run.smithi046.stderr:Adding key to root@localhost authorized_keys... 2024-01-21T08:04:57.671 INFO:teuthology.orchestra.run.smithi046.stderr:Adding host smithi046... 2024-01-21T08:04:57.672 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi046 172.21.15.46 2024-01-21T08:04:58.040 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.030+0000 7f9fbfe9b700 1 Processor -- start 2024-01-21T08:04:58.040 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.030+0000 7f9fbfe9b700 1 -- start start 2024-01-21T08:04:58.040 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.030+0000 7f9fbfe9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.030+0000 7f9fbfe9b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb80fb490 con 0x7f9fb80fab80 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.030+0000 7f9fbdc37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.031+0000 7f9fbdc37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50780/0 (socket says 172.21.15.46:50780) 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.031+0000 7f9fbdc37700 1 -- 172.21.15.46:0/523568705 learned_addr learned my addr 172.21.15.46:0/523568705 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.031+0000 7f9fbdc37700 1 -- 172.21.15.46:0/523568705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb80fb5d0 con 0x7f9fb80fab80 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.031+0000 7f9fbdc37700 1 --2- 172.21.15.46:0/523568705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb80fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9fa800b5f0 tx=0x7f9fa8006ee0).ready entity=mon.0 client_cookie=50f6f84bc874039e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.031+0000 7f9fbcc35700 1 -- 172.21.15.46:0/523568705 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa800d600 con 0x7f9fb80fab80 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.031+0000 7f9fbcc35700 1 -- 172.21.15.46:0/523568705 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9fa800dbd0 con 0x7f9fb80fab80 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.031+0000 7f9fbcc35700 1 -- 172.21.15.46:0/523568705 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa8011c00 con 0x7f9fb80fab80 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.032+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/523568705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 msgr2=0x7f9fb80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.032+0000 7f9fbfe9b700 1 --2- 172.21.15.46:0/523568705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb80fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9fa800b5f0 tx=0x7f9fa8006ee0).stop 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.032+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/523568705 shutdown_connections 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.032+0000 7f9fbfe9b700 1 --2- 172.21.15.46:0/523568705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb80fafa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.032+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/523568705 >> 172.21.15.46:0/523568705 conn(0x7f9fb80f63c0 msgr2=0x7f9fb80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.032+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/523568705 shutdown_connections 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.032+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/523568705 wait complete. 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbfe9b700 1 Processor -- start 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbfe9b700 1 -- start start 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbfe9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb81098e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbfe9b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb8109dd0 con 0x7f9fb80fab80 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbdc37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb81098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbdc37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb81098e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50788/0 (socket says 172.21.15.46:50788) 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbdc37700 1 -- 172.21.15.46:0/3647890684 learned_addr learned my addr 172.21.15.46:0/3647890684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbdc37700 1 -- 172.21.15.46:0/3647890684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa800b200 con 0x7f9fb80fab80 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fbdc37700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb81098e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9fa800b1d0 tx=0x7f9fa8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.033+0000 7f9fa67fc700 1 -- 172.21.15.46:0/3647890684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa801b410 con 0x7f9fb80fab80 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.034+0000 7f9fa67fc700 1 -- 172.21.15.46:0/3647890684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9fa801b9e0 con 0x7f9fb80fab80 2024-01-21T08:04:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.034+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb8109fd0 con 0x7f9fb80fab80 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.034+0000 7f9fa67fc700 1 -- 172.21.15.46:0/3647890684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa801a840 con 0x7f9fb80fab80 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.034+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb810a3e0 con 0x7f9fb80fab80 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.034+0000 7f9fa67fc700 1 -- 172.21.15.46:0/3647890684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9fa800f070 con 0x7f9fb80fab80 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.035+0000 7f9fa67fc700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f9fa002fd50 0x7f9fa0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.035+0000 7f9fa67fc700 1 -- 172.21.15.46:0/3647890684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9fa8041290 con 0x7f9fb80fab80 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.035+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb8045ad0 con 0x7f9fb80fab80 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.035+0000 7f9fbd436700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f9fa002fd50 0x7f9fa0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.035+0000 7f9fbd436700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f9fa002fd50 0x7f9fa0032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9fb40096a0 tx=0x7f9fb4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:58.044 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.039+0000 7f9fa67fc700 1 -- 172.21.15.46:0/3647890684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fa8016070 con 0x7f9fb80fab80 2024-01-21T08:04:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.221+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fb81037a0 con 0x7f9fa002fd50 2024-01-21T08:04:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:58.223907+0000 mgr.smithi046.lmxnwc (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:04:58.835 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Added host 'smithi046' 2024-01-21T08:04:58.835 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.831+0000 7f9fa67fc700 1 -- 172.21.15.46:0/3647890684 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f9fb81037a0 con 0x7f9fa002fd50 2024-01-21T08:04:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f9fa002fd50 msgr2=0x7f9fa0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f9fa002fd50 0x7f9fa0032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9fb40096a0 tx=0x7f9fb4006b40).stop 2024-01-21T08:04:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 msgr2=0x7f9fb81098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb81098e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9fa800b1d0 tx=0x7f9fa8006ee0).stop 2024-01-21T08:04:58.838 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 shutdown_connections 2024-01-21T08:04:58.838 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f9fa002fd50 0x7f9fa0032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:58.838 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 --2- 172.21.15.46:0/3647890684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb80fab80 0x7f9fb81098e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:58.838 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 >> 172.21.15.46:0/3647890684 conn(0x7f9fb80f63c0 msgr2=0x7f9fb80f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:58.838 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 shutdown_connections 2024-01-21T08:04:58.838 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:58.834+0000 7f9fbfe9b700 1 -- 172.21.15.46:0/3647890684 wait complete. 2024-01-21T08:04:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:Deploying mon service with default placement... 2024-01-21T08:04:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-21T08:04:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.456+0000 7f0cdc090700 1 Processor -- start 2024-01-21T08:04:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.456+0000 7f0cdc090700 1 -- start start 2024-01-21T08:04:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.456+0000 7f0cdc090700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.456+0000 7f0cdc090700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40fedd0 con 0x7f0cd4070430 2024-01-21T08:04:59.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd9e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:59.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd9e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50794/0 (socket says 172.21.15.46:50794) 2024-01-21T08:04:59.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd9e2c700 1 -- 172.21.15.46:0/1686247506 learned_addr learned my addr 172.21.15.46:0/1686247506 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:59.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd9e2c700 1 -- 172.21.15.46:0/1686247506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40fef10 con 0x7f0cd4070430 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd9e2c700 1 --2- 172.21.15.46:0/1686247506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd40fe8e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0cc400b5f0 tx=0x7f0cc4006ee0).ready entity=mon.0 client_cookie=b34eca69a65d441c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd8e2a700 1 -- 172.21.15.46:0/1686247506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc400d600 con 0x7f0cd4070430 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd8e2a700 1 -- 172.21.15.46:0/1686247506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cc400dbd0 con 0x7f0cd4070430 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.457+0000 7f0cd8e2a700 1 -- 172.21.15.46:0/1686247506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc4011c00 con 0x7f0cd4070430 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.458+0000 7f0cdc090700 1 -- 172.21.15.46:0/1686247506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 msgr2=0x7f0cd40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.458+0000 7f0cdc090700 1 --2- 172.21.15.46:0/1686247506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd40fe8e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0cc400b5f0 tx=0x7f0cc4006ee0).stop 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.458+0000 7f0cdc090700 1 -- 172.21.15.46:0/1686247506 shutdown_connections 2024-01-21T08:04:59.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.458+0000 7f0cdc090700 1 --2- 172.21.15.46:0/1686247506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd40fe8e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.458+0000 7f0cdc090700 1 -- 172.21.15.46:0/1686247506 >> 172.21.15.46:0/1686247506 conn(0x7f0cd40f63c0 msgr2=0x7f0cd40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.458+0000 7f0cdc090700 1 -- 172.21.15.46:0/1686247506 shutdown_connections 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.458+0000 7f0cdc090700 1 -- 172.21.15.46:0/1686247506 wait complete. 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.459+0000 7f0cdc090700 1 Processor -- start 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.459+0000 7f0cdc090700 1 -- start start 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.459+0000 7f0cdc090700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.459+0000 7f0cdc090700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd410af30 con 0x7f0cd4070430 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.459+0000 7f0cd9e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.459+0000 7f0cd9e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50798/0 (socket says 172.21.15.46:50798) 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.459+0000 7f0cd9e2c700 1 -- 172.21.15.46:0/1155495007 learned_addr learned my addr 172.21.15.46:0/1155495007 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.460+0000 7f0cd9e2c700 1 -- 172.21.15.46:0/1155495007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc400b200 con 0x7f0cd4070430 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.460+0000 7f0cd9e2c700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd410aa40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0cc400e040 tx=0x7f0cc4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:59.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.460+0000 7f0cc27fc700 1 -- 172.21.15.46:0/1155495007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc400d9f0 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.460+0000 7f0cc27fc700 1 -- 172.21.15.46:0/1155495007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cc401a9f0 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.460+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd410b130 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.460+0000 7f0cc27fc700 1 -- 172.21.15.46:0/1155495007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc401b8c0 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.460+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd410b540 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.461+0000 7f0cc27fc700 1 -- 172.21.15.46:0/1155495007 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0cc400f070 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.461+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd4045ad0 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.461+0000 7f0cc27fc700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f0cbc02fdb0 0x7f0cbc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.461+0000 7f0cc27fc700 1 -- 172.21.15.46:0/1155495007 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0cc4041270 con 0x7f0cd4070430 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.461+0000 7f0cd962b700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f0cbc02fdb0 0x7f0cbc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:04:59.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.462+0000 7f0cd962b700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f0cbc02fdb0 0x7f0cbc032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0cd00096a0 tx=0x7f0cd0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:04:59.468 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.466+0000 7f0cc27fc700 1 -- 172.21.15.46:0/1155495007 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cc4022310 con 0x7f0cd4070430 2024-01-21T08:04:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.655+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cd410be80 con 0x7f0cbc02fdb0 2024-01-21T08:04:59.662 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.659+0000 7f0cc27fc700 1 -- 172.21.15.46:0/1155495007 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0cd410be80 con 0x7f0cbc02fdb0 2024-01-21T08:04:59.662 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-21T08:04:59.664 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f0cbc02fdb0 msgr2=0x7f0cbc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:59.664 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f0cbc02fdb0 0x7f0cbc032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0cd00096a0 tx=0x7f0cd0006b40).stop 2024-01-21T08:04:59.664 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 msgr2=0x7f0cd410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:04:59.664 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd410aa40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0cc400e040 tx=0x7f0cc4006ee0).stop 2024-01-21T08:04:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 shutdown_connections 2024-01-21T08:04:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f0cbc02fdb0 0x7f0cbc032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 --2- 172.21.15.46:0/1155495007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4070430 0x7f0cd410aa40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:04:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 >> 172.21.15.46:0/1155495007 conn(0x7f0cd40f63c0 msgr2=0x7f0cd40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:04:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 shutdown_connections 2024-01-21T08:04:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:04:59.661+0000 7f0cdc090700 1 -- 172.21.15.46:0/1155495007 wait complete. 2024-01-21T08:04:59.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:58.832648+0000 mon.smithi046 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:04:59.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:04:58.832985+0000 mgr.smithi046.lmxnwc (mgr.14116) 9 : cephadm [INF] Added host smithi046 2024-01-21T08:04:59.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:58.833229+0000 mon.smithi046 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:04:59.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:04:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:59.660913+0000 mon.smithi046 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:00.010 INFO:teuthology.orchestra.run.smithi046.stderr:Deploying mgr service with default placement... 2024-01-21T08:05:00.010 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-21T08:05:00.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.396+0000 7f1eef8de700 1 Processor -- start 2024-01-21T08:05:00.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.397+0000 7f1eef8de700 1 -- start start 2024-01-21T08:05:00.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.397+0000 7f1eef8de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee80fe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:00.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.397+0000 7f1eef8de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee80fedf0 con 0x7f1ee8070430 2024-01-21T08:05:00.400 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.397+0000 7f1eed67a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee80fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.397+0000 7f1eed67a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee80fe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50810/0 (socket says 172.21.15.46:50810) 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.397+0000 7f1eed67a700 1 -- 172.21.15.46:0/1055497571 learned_addr learned my addr 172.21.15.46:0/1055497571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.398+0000 7f1eed67a700 1 -- 172.21.15.46:0/1055497571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee80fef30 con 0x7f1ee8070430 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.398+0000 7f1eed67a700 1 --2- 172.21.15.46:0/1055497571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee80fe900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f1ed800b5f0 tx=0x7f1ed8006ee0).ready entity=mon.0 client_cookie=8ea10ee7a5312343 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.398+0000 7f1ed7fff700 1 -- 172.21.15.46:0/1055497571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed800d600 con 0x7f1ee8070430 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.398+0000 7f1ed7fff700 1 -- 172.21.15.46:0/1055497571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ed800dbd0 con 0x7f1ee8070430 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.398+0000 7f1ed7fff700 1 -- 172.21.15.46:0/1055497571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed8011c00 con 0x7f1ee8070430 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 -- 172.21.15.46:0/1055497571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 msgr2=0x7f1ee80fe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 --2- 172.21.15.46:0/1055497571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee80fe900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f1ed800b5f0 tx=0x7f1ed8006ee0).stop 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 -- 172.21.15.46:0/1055497571 shutdown_connections 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 --2- 172.21.15.46:0/1055497571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee80fe900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:00.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 -- 172.21.15.46:0/1055497571 >> 172.21.15.46:0/1055497571 conn(0x7f1ee80f63c0 msgr2=0x7f1ee80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 -- 172.21.15.46:0/1055497571 shutdown_connections 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 -- 172.21.15.46:0/1055497571 wait complete. 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 Processor -- start 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.399+0000 7f1eef8de700 1 -- start start 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eef8de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee810aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eef8de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee810af40 con 0x7f1ee8070430 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eed67a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee810aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eed67a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee810aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50812/0 (socket says 172.21.15.46:50812) 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eed67a700 1 -- 172.21.15.46:0/3030275652 learned_addr learned my addr 172.21.15.46:0/3030275652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eed67a700 1 -- 172.21.15.46:0/3030275652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed800b200 con 0x7f1ee8070430 2024-01-21T08:05:00.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eed67a700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee810aa50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1ed800e040 tx=0x7f1ed8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:00.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1ed5ffb700 1 -- 172.21.15.46:0/3030275652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed800d9f0 con 0x7f1ee8070430 2024-01-21T08:05:00.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee810b140 con 0x7f1ee8070430 2024-01-21T08:05:00.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1ed5ffb700 1 -- 172.21.15.46:0/3030275652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ed801a9f0 con 0x7f1ee8070430 2024-01-21T08:05:00.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.400+0000 7f1ed5ffb700 1 -- 172.21.15.46:0/3030275652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed801b8c0 con 0x7f1ee8070430 2024-01-21T08:05:00.406 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.401+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee810b550 con 0x7f1ee8070430 2024-01-21T08:05:00.406 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.401+0000 7f1ed5ffb700 1 -- 172.21.15.46:0/3030275652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1ed800f070 con 0x7f1ee8070430 2024-01-21T08:05:00.406 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.402+0000 7f1ed5ffb700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f1ed002fd50 0x7f1ed0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:00.406 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.402+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee8045ad0 con 0x7f1ee8070430 2024-01-21T08:05:00.406 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.402+0000 7f1ed5ffb700 1 -- 172.21.15.46:0/3030275652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1ed8041320 con 0x7f1ee8070430 2024-01-21T08:05:00.406 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.402+0000 7f1eece79700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f1ed002fd50 0x7f1ed0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:00.406 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.402+0000 7f1eece79700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f1ed002fd50 0x7f1ed0032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f1ee40096a0 tx=0x7f1ee4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:00.408 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.406+0000 7f1ed5ffb700 1 -- 172.21.15.46:0/3030275652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ed8016020 con 0x7f1ee8070430 2024-01-21T08:05:00.596 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.593+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ee810be90 con 0x7f1ed002fd50 2024-01-21T08:05:00.599 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.596+0000 7f1ed5ffb700 1 -- 172.21.15.46:0/3030275652 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1ee810be90 con 0x7f1ed002fd50 2024-01-21T08:05:00.599 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-21T08:05:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f1ed002fd50 msgr2=0x7f1ed0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f1ed002fd50 0x7f1ed0032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f1ee40096a0 tx=0x7f1ee4006b40).stop 2024-01-21T08:05:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 msgr2=0x7f1ee810aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee810aa50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1ed800e040 tx=0x7f1ed8006ee0).stop 2024-01-21T08:05:00.602 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 shutdown_connections 2024-01-21T08:05:00.602 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7f1ed002fd50 0x7f1ed0032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:00.602 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 --2- 172.21.15.46:0/3030275652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ee8070430 0x7f1ee810aa50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:00.602 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.598+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 >> 172.21.15.46:0/3030275652 conn(0x7f1ee80f63c0 msgr2=0x7f1ee80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:00.602 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.599+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 shutdown_connections 2024-01-21T08:05:00.602 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:00.599+0000 7f1eef8de700 1 -- 172.21.15.46:0/3030275652 wait complete. 2024-01-21T08:05:00.943 INFO:teuthology.orchestra.run.smithi046.stderr:Deploying crash service with default placement... 2024-01-21T08:05:00.944 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-21T08:05:00.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:04:59.658109+0000 mgr.smithi046.lmxnwc (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:00.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:04:59.658838+0000 mgr.smithi046.lmxnwc (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-21T08:05:00.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:00.598027+0000 mon.smithi046 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:01.273 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.270+0000 7feb2bd30700 1 Processor -- start 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.270+0000 7feb2bd30700 1 -- start start 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.270+0000 7feb2bd30700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.270+0000 7feb2bd30700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb240fedd0 con 0x7feb24070430 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.270+0000 7feb29acc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.270+0000 7feb29acc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52362/0 (socket says 172.21.15.46:52362) 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.270+0000 7feb29acc700 1 -- 172.21.15.46:0/1509226153 learned_addr learned my addr 172.21.15.46:0/1509226153 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.271+0000 7feb29acc700 1 -- 172.21.15.46:0/1509226153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240fef10 con 0x7feb24070430 2024-01-21T08:05:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.271+0000 7feb29acc700 1 --2- 172.21.15.46:0/1509226153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7feb2000b5f0 tx=0x7feb20006ee0).ready entity=mon.0 client_cookie=566bd51d293b1246 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.271+0000 7feb28aca700 1 -- 172.21.15.46:0/1509226153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb2000d600 con 0x7feb24070430 2024-01-21T08:05:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.271+0000 7feb28aca700 1 -- 172.21.15.46:0/1509226153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb2000dbd0 con 0x7feb24070430 2024-01-21T08:05:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.271+0000 7feb28aca700 1 -- 172.21.15.46:0/1509226153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb20011c00 con 0x7feb24070430 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 -- 172.21.15.46:0/1509226153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 msgr2=0x7feb240fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 --2- 172.21.15.46:0/1509226153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7feb2000b5f0 tx=0x7feb20006ee0).stop 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 -- 172.21.15.46:0/1509226153 shutdown_connections 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 --2- 172.21.15.46:0/1509226153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fe8e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 -- 172.21.15.46:0/1509226153 >> 172.21.15.46:0/1509226153 conn(0x7feb240f63c0 msgr2=0x7feb240f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 -- 172.21.15.46:0/1509226153 shutdown_connections 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 -- 172.21.15.46:0/1509226153 wait complete. 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 Processor -- start 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.272+0000 7feb2bd30700 1 -- start start 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb2bd30700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb2bd30700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb240fa7e0 con 0x7feb24070430 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb29acc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb29acc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52374/0 (socket says 172.21.15.46:52374) 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb29acc700 1 -- 172.21.15.46:0/625083033 learned_addr learned my addr 172.21.15.46:0/625083033 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb29acc700 1 -- 172.21.15.46:0/625083033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2000b200 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb29acc700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7feb2000e040 tx=0x7feb20006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb127fc700 1 -- 172.21.15.46:0/625083033 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb2000d9f0 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb127fc700 1 -- 172.21.15.46:0/625083033 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb2001a9f0 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb240fa980 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb127fc700 1 -- 172.21.15.46:0/625083033 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb2001b8c0 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.273+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb240fad90 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.274+0000 7feb127fc700 1 -- 172.21.15.46:0/625083033 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7feb2000f070 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.274+0000 7feb127fc700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb0c02fdb0 0x7feb0c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.274+0000 7feb127fc700 1 -- 172.21.15.46:0/625083033 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7feb20041290 con 0x7feb24070430 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.274+0000 7feb292cb700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb0c02fdb0 0x7feb0c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.275+0000 7feb292cb700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb0c02fdb0 0x7feb0c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7feb140096a0 tx=0x7feb14006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.275+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb24059100 con 0x7feb24070430 2024-01-21T08:05:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.279+0000 7feb127fc700 1 -- 172.21.15.46:0/625083033 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb2001a580 con 0x7feb24070430 2024-01-21T08:05:01.469 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.466+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7feb240ffb90 con 0x7feb0c02fdb0 2024-01-21T08:05:01.473 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.470+0000 7feb127fc700 1 -- 172.21.15.46:0/625083033 <== mgr.14116 v2:172.21.15.46:6800/2245961460 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7feb240ffb90 con 0x7feb0c02fdb0 2024-01-21T08:05:01.473 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-21T08:05:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb0c02fdb0 msgr2=0x7feb0c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb0c02fdb0 0x7feb0c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7feb140096a0 tx=0x7feb14006b40).stop 2024-01-21T08:05:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 msgr2=0x7feb240fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7feb2000e040 tx=0x7feb20006ee0).stop 2024-01-21T08:05:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 shutdown_connections 2024-01-21T08:05:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7feb0c02fdb0 0x7feb0c032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 --2- 172.21.15.46:0/625083033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb24070430 0x7feb240fc360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:01.477 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.472+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 >> 172.21.15.46:0/625083033 conn(0x7feb240f63c0 msgr2=0x7feb240f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:01.477 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.473+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 shutdown_connections 2024-01-21T08:05:01.477 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:01.473+0000 7feb2bd30700 1 -- 172.21.15.46:0/625083033 wait complete. 2024-01-21T08:05:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:Enabling mgr prometheus module... 2024-01-21T08:05:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-21T08:05:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.153+0000 7fb46852a700 1 Processor -- start 2024-01-21T08:05:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.153+0000 7fb46852a700 1 -- start start 2024-01-21T08:05:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.154+0000 7fb46852a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb460100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.154+0000 7fb46852a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4600fc730 con 0x7fb4600703d0 2024-01-21T08:05:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.154+0000 7fb4662c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb460100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.154+0000 7fb4662c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb460100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52378/0 (socket says 172.21.15.46:52378) 2024-01-21T08:05:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.154+0000 7fb4662c6700 1 -- 172.21.15.46:0/837084774 learned_addr learned my addr 172.21.15.46:0/837084774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.154+0000 7fb4662c6700 1 -- 172.21.15.46:0/837084774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb460101230 con 0x7fb4600703d0 2024-01-21T08:05:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.154+0000 7fb4662c6700 1 --2- 172.21.15.46:0/837084774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb460100d40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb45c00b5f0 tx=0x7fb45c006ee0).ready entity=mon.0 client_cookie=68739b2759d181a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.155+0000 7fb4652c4700 1 -- 172.21.15.46:0/837084774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c00d600 con 0x7fb4600703d0 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.155+0000 7fb4652c4700 1 -- 172.21.15.46:0/837084774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb45c00dbd0 con 0x7fb4600703d0 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.155+0000 7fb4652c4700 1 -- 172.21.15.46:0/837084774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c011c00 con 0x7fb4600703d0 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.155+0000 7fb46852a700 1 -- 172.21.15.46:0/837084774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 msgr2=0x7fb460100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.155+0000 7fb46852a700 1 --2- 172.21.15.46:0/837084774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb460100d40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb45c00b5f0 tx=0x7fb45c006ee0).stop 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.156+0000 7fb46852a700 1 -- 172.21.15.46:0/837084774 shutdown_connections 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.156+0000 7fb46852a700 1 --2- 172.21.15.46:0/837084774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb460100d40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.156+0000 7fb46852a700 1 -- 172.21.15.46:0/837084774 >> 172.21.15.46:0/837084774 conn(0x7fb4600f63c0 msgr2=0x7fb4600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.156+0000 7fb46852a700 1 -- 172.21.15.46:0/837084774 shutdown_connections 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.156+0000 7fb46852a700 1 -- 172.21.15.46:0/837084774 wait complete. 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.156+0000 7fb46852a700 1 Processor -- start 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.156+0000 7fb46852a700 1 -- start start 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb46852a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb46010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb46852a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46010d0a0 con 0x7fb4600703d0 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb4662c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb46010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb4662c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb46010cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52384/0 (socket says 172.21.15.46:52384) 2024-01-21T08:05:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb4662c6700 1 -- 172.21.15.46:0/1732532583 learned_addr learned my addr 172.21.15.46:0/1732532583 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb4662c6700 1 -- 172.21.15.46:0/1732532583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c00b200 con 0x7fb4600703d0 2024-01-21T08:05:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb4662c6700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb46010cbb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fb45c00b190 tx=0x7fb45c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c01b410 con 0x7fb4600703d0 2024-01-21T08:05:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46010d2a0 con 0x7fb4600703d0 2024-01-21T08:05:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb45c01b9e0 con 0x7fb4600703d0 2024-01-21T08:05:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c01a880 con 0x7fb4600703d0 2024-01-21T08:05:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.157+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46010d6b0 con 0x7fb4600703d0 2024-01-21T08:05:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.158+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb45c00f070 con 0x7fb4600703d0 2024-01-21T08:05:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.158+0000 7fb44effd700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fb44802fd50 0x7fb448032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.158+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb45c0412a0 con 0x7fb4600703d0 2024-01-21T08:05:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.158+0000 7fb465ac5700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fb44802fd50 0x7fb448032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.159+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb460059100 con 0x7fb4600703d0 2024-01-21T08:05:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.159+0000 7fb465ac5700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fb44802fd50 0x7fb448032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb4500096a0 tx=0x7fb450006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.163+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb45c016070 con 0x7fb4600703d0 2024-01-21T08:05:02.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:00.595479+0000 mgr.smithi046.lmxnwc (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:02.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:00.595997+0000 mgr.smithi046.lmxnwc (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-21T08:05:02.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:01.472028+0000 mon.smithi046 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.46:0/550702402' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:02.363 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.360+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fb460045ad0 con 0x7fb4600703d0 2024-01-21T08:05:02.836 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.833+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fb45c022d80 con 0x7fb4600703d0 2024-01-21T08:05:02.837 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.834+0000 7fb44effd700 1 -- 172.21.15.46:0/1732532583 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fb45c019300 con 0x7fb4600703d0 2024-01-21T08:05:02.841 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.837+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fb44802fd50 msgr2=0x7fb448032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:02.841 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.837+0000 7fb46852a700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fb44802fd50 0x7fb448032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb4500096a0 tx=0x7fb450006b40).stop 2024-01-21T08:05:02.841 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.837+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 msgr2=0x7fb46010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:02.841 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.837+0000 7fb46852a700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb46010cbb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fb45c00b190 tx=0x7fb45c006ee0).stop 2024-01-21T08:05:02.841 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.837+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 shutdown_connections 2024-01-21T08:05:02.841 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.837+0000 7fb46852a700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fb44802fd50 0x7fb448032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:02.842 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.837+0000 7fb46852a700 1 --2- 172.21.15.46:0/1732532583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4600703d0 0x7fb46010cbb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:02.842 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.838+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 >> 172.21.15.46:0/1732532583 conn(0x7fb4600f63c0 msgr2=0x7fb4600f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:02.842 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.838+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 shutdown_connections 2024-01-21T08:05:02.842 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:02.838+0000 7fb46852a700 1 -- 172.21.15.46:0/1732532583 wait complete. 2024-01-21T08:05:03.129 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:01.469178+0000 mgr.smithi046.lmxnwc (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:03.129 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:01.469813+0000 mgr.smithi046.lmxnwc (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-21T08:05:03.129 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:02.362285+0000 mon.smithi046 (mon.0) 60 : audit [INF] from='client.? 172.21.15.46:0/1732532583' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-21T08:05:03.202 INFO:teuthology.orchestra.run.smithi046.stderr:Deploying prometheus service with default placement... 2024-01-21T08:05:03.202 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-21T08:05:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.562+0000 7fdec4e07700 1 Processor -- start 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdec4e07700 1 -- start start 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdec4e07700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec0100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdec4e07700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec00fc750 con 0x7fdec00703b0 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdebe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec0100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdebe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec0100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52408/0 (socket says 172.21.15.46:52408) 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdebe59c700 1 -- 172.21.15.46:0/2241508308 learned_addr learned my addr 172.21.15.46:0/2241508308 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdebe59c700 1 -- 172.21.15.46:0/2241508308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec0101250 con 0x7fdec00703b0 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.563+0000 7fdebe59c700 1 --2- 172.21.15.46:0/2241508308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec0100d60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fdeb000b5f0 tx=0x7fdeb0006ee0).ready entity=mon.0 client_cookie=996e8d6cec56116f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdebd59a700 1 -- 172.21.15.46:0/2241508308 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeb000d600 con 0x7fdec00703b0 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdebd59a700 1 -- 172.21.15.46:0/2241508308 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdeb000dbd0 con 0x7fdec00703b0 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdebd59a700 1 -- 172.21.15.46:0/2241508308 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeb0011c00 con 0x7fdec00703b0 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdec4e07700 1 -- 172.21.15.46:0/2241508308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 msgr2=0x7fdec0100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdec4e07700 1 --2- 172.21.15.46:0/2241508308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec0100d60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fdeb000b5f0 tx=0x7fdeb0006ee0).stop 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdec4e07700 1 -- 172.21.15.46:0/2241508308 shutdown_connections 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdec4e07700 1 --2- 172.21.15.46:0/2241508308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec0100d60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdec4e07700 1 -- 172.21.15.46:0/2241508308 >> 172.21.15.46:0/2241508308 conn(0x7fdec00f63c0 msgr2=0x7fdec00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdec4e07700 1 -- 172.21.15.46:0/2241508308 shutdown_connections 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.564+0000 7fdec4e07700 1 -- 172.21.15.46:0/2241508308 wait complete. 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.565+0000 7fdec4e07700 1 Processor -- start 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.565+0000 7fdec4e07700 1 -- start start 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.565+0000 7fdec4e07700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.565+0000 7fdec4e07700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec010d1e0 con 0x7fdec00703b0 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.565+0000 7fdebe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.565+0000 7fdebe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52410/0 (socket says 172.21.15.46:52410) 2024-01-21T08:05:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.565+0000 7fdebe59c700 1 -- 172.21.15.46:0/3354183390 learned_addr learned my addr 172.21.15.46:0/3354183390 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.566+0000 7fdebe59c700 1 -- 172.21.15.46:0/3354183390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb000b200 con 0x7fdec00703b0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.566+0000 7fdebe59c700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec010ccf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fdeb000e040 tx=0x7fdeb0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.566+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeb000d9f0 con 0x7fdec00703b0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.566+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdeb001a9f0 con 0x7fdec00703b0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.566+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeb001b8c0 con 0x7fdec00703b0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.566+0000 7fdec4e07700 1 -- 172.21.15.46:0/3354183390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec010d3e0 con 0x7fdec00703b0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.566+0000 7fdec4e07700 1 -- 172.21.15.46:0/3354183390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec010d7f0 con 0x7fdec00703b0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.567+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea0004fa0 con 0x7fdec00703b0 2024-01-21T08:05:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.568+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fdeb000f070 con 0x7fdec00703b0 2024-01-21T08:05:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.568+0000 7fdeaeffd700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.568+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fdeb0041a80 con 0x7fdec00703b0 2024-01-21T08:05:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.568+0000 7fdebdd9b700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 msgr2=0x7fdea4032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2245961460 2024-01-21T08:05:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.568+0000 7fdebdd9b700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T08:05:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.572+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdeb002d7a0 con 0x7fdec00703b0 2024-01-21T08:05:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.759+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 --> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fdea0000bc0 con 0x7fdea402fe50 2024-01-21T08:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.769+0000 7fdebdd9b700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 msgr2=0x7fdea4032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2245961460 2024-01-21T08:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:03.769+0000 7fdebdd9b700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T08:05:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:04.170+0000 7fdebdd9b700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 msgr2=0x7fdea4032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2245961460 2024-01-21T08:05:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:04.170+0000 7fdebdd9b700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T08:05:04.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:02.836345+0000 mon.smithi046 (mon.0) 61 : audit [INF] from='client.? 172.21.15.46:0/1732532583' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-21T08:05:04.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:02.836412+0000 mon.smithi046 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi046.lmxnwc(active, since 16s) 2024-01-21T08:05:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:04.971+0000 7fdebdd9b700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 msgr2=0x7fdea4032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2245961460 2024-01-21T08:05:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:04.971+0000 7fdebdd9b700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T08:05:06.576 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:06.572+0000 7fdebdd9b700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 msgr2=0x7fdea4032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2245961460 2024-01-21T08:05:06.576 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:06.572+0000 7fdebdd9b700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T08:05:08.075 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:08.072+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fdeb0024c90 con 0x7fdec00703b0 2024-01-21T08:05:08.075 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:08.072+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 msgr2=0x7fdea4032310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T08:05:08.075 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:08.072+0000 7fdeaeffd700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:05:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.073+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fdeb0019d80 con 0x7fdec00703b0 2024-01-21T08:05:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.073+0000 7fdeaeffd700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fdea4033c70 0x7fdea4036060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.073+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 --> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fdea0000bc0 con 0x7fdea4033c70 2024-01-21T08:05:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.074+0000 7fdebdd9b700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fdea4033c70 0x7fdea4036060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.074+0000 7fdebdd9b700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fdea4033c70 0x7fdea4036060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdeb40036d0 tx=0x7fdeb4006d60).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.079+0000 7fdeaeffd700 1 -- 172.21.15.46:0/3354183390 <== mgr.14144 v2:172.21.15.46:6800/1810147681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fdea0000bc0 con 0x7fdea4033c70 2024-01-21T08:05:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-21T08:05:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fdea4033c70 msgr2=0x7fdea4036060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fdea4033c70 0x7fdea4036060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdeb40036d0 tx=0x7fdeb4006d60).stop 2024-01-21T08:05:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 msgr2=0x7fdec010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec010ccf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fdeb000e040 tx=0x7fdeb0006ee0).stop 2024-01-21T08:05:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 shutdown_connections 2024-01-21T08:05:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00703b0 0x7fdec010ccf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fdea4033c70 0x7fdea4036060 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 --2- 172.21.15.46:0/3354183390 >> [v2:172.21.15.46:6800/2245961460,v1:172.21.15.46:6801/2245961460] conn(0x7fdea402fe50 0x7fdea4032310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:05:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.081+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 >> 172.21.15.46:0/3354183390 conn(0x7fdec00f63c0 msgr2=0x7fdec00f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.082+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 shutdown_connections 2024-01-21T08:05:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.082+0000 7fdeacff9700 1 -- 172.21.15.46:0/3354183390 wait complete. 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:08.021663+0000 mon.smithi046 (mon.0) 63 : cluster [INF] Active manager daemon smithi046.lmxnwc restarted 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:08.022098+0000 mon.smithi046 (mon.0) 64 : cluster [INF] Activating manager daemon smithi046.lmxnwc 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:08.023649+0000 mon.smithi046 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:08.074625+0000 mon.smithi046 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi046.lmxnwc(active, starting, since 0.0526358s) 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.075690+0000 mon.smithi046 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.076334+0000 mon.smithi046 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr metadata", "who": "smithi046.lmxnwc", "id": "smithi046.lmxnwc"}]: dispatch 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.077253+0000 mon.smithi046 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.077402+0000 mon.smithi046 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.077534+0000 mon.smithi046 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:08.090940+0000 mon.smithi046 (mon.0) 72 : cluster [INF] Manager daemon smithi046.lmxnwc is now available 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.193990+0000 mon.smithi046 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:05:09.362 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.195636+0000 mon.smithi046 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:05:09.363 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.217290+0000 mon.smithi046 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:05:09.363 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:08.221473+0000 mon.smithi046 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:05:09.380 INFO:teuthology.orchestra.run.smithi046.stderr:Deploying grafana service with default placement... 2024-01-21T08:05:09.381 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-21T08:05:09.738 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.734+0000 7f385fb78700 1 Processor -- start 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.734+0000 7f385fb78700 1 -- start start 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.735+0000 7f385fb78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f3858100d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.735+0000 7f385fb78700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38580fc780 con 0x7f3858070400 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.735+0000 7f385d914700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f3858100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.735+0000 7f385d914700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f3858100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52464/0 (socket says 172.21.15.46:52464) 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.735+0000 7f385d914700 1 -- 172.21.15.46:0/1321878141 learned_addr learned my addr 172.21.15.46:0/1321878141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.735+0000 7f385d914700 1 -- 172.21.15.46:0/1321878141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3858101280 con 0x7f3858070400 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.735+0000 7f385d914700 1 --2- 172.21.15.46:0/1321878141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f3858100d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f385400b5f0 tx=0x7f3854006ee0).ready entity=mon.0 client_cookie=8c6d83baa212deff server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385c912700 1 -- 172.21.15.46:0/1321878141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f385400d600 con 0x7f3858070400 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385c912700 1 -- 172.21.15.46:0/1321878141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f385400dbd0 con 0x7f3858070400 2024-01-21T08:05:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385c912700 1 -- 172.21.15.46:0/1321878141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3854011c00 con 0x7f3858070400 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385fb78700 1 -- 172.21.15.46:0/1321878141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 msgr2=0x7f3858100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385fb78700 1 --2- 172.21.15.46:0/1321878141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f3858100d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f385400b5f0 tx=0x7f3854006ee0).stop 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385fb78700 1 -- 172.21.15.46:0/1321878141 shutdown_connections 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385fb78700 1 --2- 172.21.15.46:0/1321878141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f3858100d90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385fb78700 1 -- 172.21.15.46:0/1321878141 >> 172.21.15.46:0/1321878141 conn(0x7f38580f63b0 msgr2=0x7f38580f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385fb78700 1 -- 172.21.15.46:0/1321878141 shutdown_connections 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.736+0000 7f385fb78700 1 -- 172.21.15.46:0/1321878141 wait complete. 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385fb78700 1 Processor -- start 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385fb78700 1 -- start start 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385fb78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f385810cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385fb78700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385810d0e0 con 0x7f3858070400 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385d914700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f385810cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:09.742 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385d914700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f385810cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52476/0 (socket says 172.21.15.46:52476) 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385d914700 1 -- 172.21.15.46:0/737959804 learned_addr learned my addr 172.21.15.46:0/737959804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.737+0000 7f385d914700 1 -- 172.21.15.46:0/737959804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385400b200 con 0x7f3858070400 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.738+0000 7f385d914700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f385810cbf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f385400b190 tx=0x7f3854006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.738+0000 7f38467fc700 1 -- 172.21.15.46:0/737959804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f385401b410 con 0x7f3858070400 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.738+0000 7f38467fc700 1 -- 172.21.15.46:0/737959804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f385401b9e0 con 0x7f3858070400 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.738+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385810d2e0 con 0x7f3858070400 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.738+0000 7f38467fc700 1 -- 172.21.15.46:0/737959804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f385401a880 con 0x7f3858070400 2024-01-21T08:05:09.743 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.738+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385810d6f0 con 0x7f3858070400 2024-01-21T08:05:09.744 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.739+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3858045ad0 con 0x7f3858070400 2024-01-21T08:05:09.745 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.739+0000 7f38467fc700 1 -- 172.21.15.46:0/737959804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f385400f070 con 0x7f3858070400 2024-01-21T08:05:09.745 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.739+0000 7f38467fc700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f384002fd50 0x7f3840032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:09.745 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.739+0000 7f38467fc700 1 -- 172.21.15.46:0/737959804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f38540411b0 con 0x7f3858070400 2024-01-21T08:05:09.745 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.739+0000 7f385d113700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f384002fd50 0x7f3840032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:09.745 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.740+0000 7f385d113700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f384002fd50 0x7f3840032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f38480096a0 tx=0x7f3848006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:09.746 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.743+0000 7f38467fc700 1 -- 172.21.15.46:0/737959804 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3854019950 con 0x7f3858070400 2024-01-21T08:05:09.936 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.933+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 --> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f385810dfd0 con 0x7f384002fd50 2024-01-21T08:05:09.939 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.937+0000 7f38467fc700 1 -- 172.21.15.46:0/737959804 <== mgr.14144 v2:172.21.15.46:6800/1810147681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f385810dfd0 con 0x7f384002fd50 2024-01-21T08:05:09.940 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.938+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f384002fd50 msgr2=0x7f3840032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.938+0000 7f385fb78700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f384002fd50 0x7f3840032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f38480096a0 tx=0x7f3848006b40).stop 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.938+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 msgr2=0x7f385810cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.938+0000 7f385fb78700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f385810cbf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f385400b190 tx=0x7f3854006ee0).stop 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.939+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 shutdown_connections 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.939+0000 7f385fb78700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f384002fd50 0x7f3840032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.939+0000 7f385fb78700 1 --2- 172.21.15.46:0/737959804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3858070400 0x7f385810cbf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.939+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 >> 172.21.15.46:0/737959804 conn(0x7f38580f63b0 msgr2=0x7f38580f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.939+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 shutdown_connections 2024-01-21T08:05:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:09.939+0000 7f385fb78700 1 -- 172.21.15.46:0/737959804 wait complete. 2024-01-21T08:05:10.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:09.076963+0000 mon.smithi046 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi046.lmxnwc(active, since 1.05498s) 2024-01-21T08:05:10.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:09.078532+0000 mgr.smithi046.lmxnwc (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-21T08:05:10.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:09.080724+0000 mon.smithi046 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:10.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:09.938409+0000 mon.smithi046 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:Deploying node-exporter service with default placement... 2024-01-21T08:05:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.802+0000 7f916f552700 1 Processor -- start 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.802+0000 7f916f552700 1 -- start start 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.802+0000 7f916f552700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f9168100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.802+0000 7f916f552700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91680fc750 con 0x7f91680703b0 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.802+0000 7f916d2ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f9168100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.802+0000 7f916d2ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f9168100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43574/0 (socket says 172.21.15.46:43574) 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.803+0000 7f916d2ee700 1 -- 172.21.15.46:0/3401803549 learned_addr learned my addr 172.21.15.46:0/3401803549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.803+0000 7f916d2ee700 1 -- 172.21.15.46:0/3401803549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168101250 con 0x7f91680703b0 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.803+0000 7f916d2ee700 1 --2- 172.21.15.46:0/3401803549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f9168100d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f915800b5f0 tx=0x7f9158006ee0).ready entity=mon.0 client_cookie=18d6c15413023969 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.803+0000 7f9157fff700 1 -- 172.21.15.46:0/3401803549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915800d600 con 0x7f91680703b0 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.803+0000 7f9157fff700 1 -- 172.21.15.46:0/3401803549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f915800dbd0 con 0x7f91680703b0 2024-01-21T08:05:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.803+0000 7f9157fff700 1 -- 172.21.15.46:0/3401803549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9158011c00 con 0x7f91680703b0 2024-01-21T08:05:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 -- 172.21.15.46:0/3401803549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 msgr2=0x7f9168100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 --2- 172.21.15.46:0/3401803549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f9168100d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f915800b5f0 tx=0x7f9158006ee0).stop 2024-01-21T08:05:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 -- 172.21.15.46:0/3401803549 shutdown_connections 2024-01-21T08:05:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 --2- 172.21.15.46:0/3401803549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f9168100d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 -- 172.21.15.46:0/3401803549 >> 172.21.15.46:0/3401803549 conn(0x7f91680f63c0 msgr2=0x7f91680f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 -- 172.21.15.46:0/3401803549 shutdown_connections 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 -- 172.21.15.46:0/3401803549 wait complete. 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 Processor -- start 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.804+0000 7f916f552700 1 -- start start 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916f552700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f916810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916f552700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916810d1e0 con 0x7f91680703b0 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916d2ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f916810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916d2ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f916810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43584/0 (socket says 172.21.15.46:43584) 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916d2ee700 1 -- 172.21.15.46:0/231666233 learned_addr learned my addr 172.21.15.46:0/231666233 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916d2ee700 1 -- 172.21.15.46:0/231666233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915800b200 con 0x7f91680703b0 2024-01-21T08:05:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916d2ee700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f916810ccf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f915800e040 tx=0x7f9158006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915800d9f0 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f915801a9f0 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916810d3e0 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915801b8c0 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.805+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916810d7f0 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.806+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f915800f070 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.806+0000 7f9155ffb700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f915002fd70 0x7f9150032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.806+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9168045ad0 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.806+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f91580412d0 con 0x7f91680703b0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.807+0000 7f916caed700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f915002fd70 0x7f9150032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.807+0000 7f916caed700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f915002fd70 0x7f9150032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f91640096a0 tx=0x7f9164006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.811+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9158016070 con 0x7f91680703b0 2024-01-21T08:05:10.939 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.937+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f91580257e0 con 0x7f91680703b0 2024-01-21T08:05:11.001 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:10.999+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 --> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f916810ee30 con 0x7f915002fd70 2024-01-21T08:05:11.005 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.003+0000 7f9155ffb700 1 -- 172.21.15.46:0/231666233 <== mgr.14144 v2:172.21.15.46:6800/1810147681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f916810ee30 con 0x7f915002fd70 2024-01-21T08:05:11.005 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f915002fd70 msgr2=0x7f9150032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f915002fd70 0x7f9150032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f91640096a0 tx=0x7f9164006b40).stop 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 msgr2=0x7f916810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f916810ccf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f915800e040 tx=0x7f9158006ee0).stop 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 shutdown_connections 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f915002fd70 0x7f9150032230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 --2- 172.21.15.46:0/231666233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91680703b0 0x7f916810ccf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 >> 172.21.15.46:0/231666233 conn(0x7f91680f63c0 msgr2=0x7f91680f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 shutdown_connections 2024-01-21T08:05:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.004+0000 7f916f552700 1 -- 172.21.15.46:0/231666233 wait complete. 2024-01-21T08:05:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:Deploying alertmanager service with default placement... 2024-01-21T08:05:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-21T08:05:11.447 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:09.935758+0000 mgr.smithi046.lmxnwc (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:11.448 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:09.936269+0000 mgr.smithi046.lmxnwc (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-21T08:05:11.448 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:10.939836+0000 mon.smithi046 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi046.lmxnwc(active, since 2s) 2024-01-21T08:05:11.448 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:11.004279+0000 mon.smithi046 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.693+0000 7f7cea106700 1 Processor -- start 2024-01-21T08:05:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.693+0000 7f7cea106700 1 -- start start 2024-01-21T08:05:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.693+0000 7f7cea106700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce40fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.693+0000 7f7cea106700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce40fb4a0 con 0x7f7ce40fab90 2024-01-21T08:05:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.693+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce40fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.694+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce40fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43596/0 (socket says 172.21.15.46:43596) 2024-01-21T08:05:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.694+0000 7f7ce37fe700 1 -- 172.21.15.46:0/195701952 learned_addr learned my addr 172.21.15.46:0/195701952 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.694+0000 7f7ce37fe700 1 -- 172.21.15.46:0/195701952 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce40fb5e0 con 0x7f7ce40fab90 2024-01-21T08:05:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.694+0000 7f7ce37fe700 1 --2- 172.21.15.46:0/195701952 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce40fafb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7cd800b5f0 tx=0x7f7cd8006ee0).ready entity=mon.0 client_cookie=5bdb2d7bb5a4055 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.694+0000 7f7ce27fc700 1 -- 172.21.15.46:0/195701952 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd800d600 con 0x7f7ce40fab90 2024-01-21T08:05:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.694+0000 7f7ce27fc700 1 -- 172.21.15.46:0/195701952 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7cd800dbd0 con 0x7f7ce40fab90 2024-01-21T08:05:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.694+0000 7f7ce27fc700 1 -- 172.21.15.46:0/195701952 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd8011c00 con 0x7f7ce40fab90 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 -- 172.21.15.46:0/195701952 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 msgr2=0x7f7ce40fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 --2- 172.21.15.46:0/195701952 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce40fafb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7cd800b5f0 tx=0x7f7cd8006ee0).stop 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 -- 172.21.15.46:0/195701952 shutdown_connections 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 --2- 172.21.15.46:0/195701952 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce40fafb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 -- 172.21.15.46:0/195701952 >> 172.21.15.46:0/195701952 conn(0x7f7ce40f63b0 msgr2=0x7f7ce40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 -- 172.21.15.46:0/195701952 shutdown_connections 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 -- 172.21.15.46:0/195701952 wait complete. 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 Processor -- start 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.695+0000 7f7cea106700 1 -- start start 2024-01-21T08:05:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7cea106700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce410cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7cea106700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce410d100 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce410cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce410cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43606/0 (socket says 172.21.15.46:43606) 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ce37fe700 1 -- 172.21.15.46:0/3844944304 learned_addr learned my addr 172.21.15.46:0/3844944304 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ce37fe700 1 -- 172.21.15.46:0/3844944304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd800b200 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ce37fe700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce410cc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7cd800e040 tx=0x7f7cd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ccbfff700 1 -- 172.21.15.46:0/3844944304 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd801b410 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ccbfff700 1 -- 172.21.15.46:0/3844944304 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7cd801b9e0 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7ccbfff700 1 -- 172.21.15.46:0/3844944304 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd801a820 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce410d300 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.696+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce410d710 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.697+0000 7f7ccbfff700 1 -- 172.21.15.46:0/3844944304 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7cd800f070 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.697+0000 7f7ccbfff700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f7cd002fe40 0x7f7cd0032300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.697+0000 7f7ccbfff700 1 -- 172.21.15.46:0/3844944304 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7cd8041620 con 0x7f7ce40fab90 2024-01-21T08:05:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.697+0000 7f7ce2ffd700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f7cd002fe40 0x7f7cd0032300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.698+0000 7f7ce2ffd700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f7cd002fe40 0x7f7cd0032300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7cd40096a0 tx=0x7f7cd4006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.698+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce4045ad0 con 0x7f7ce40fab90 2024-01-21T08:05:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.702+0000 7f7ccbfff700 1 -- 172.21.15.46:0/3844944304 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cd8016030 con 0x7f7ce40fab90 2024-01-21T08:05:11.891 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.888+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 --> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ce41026e0 con 0x7f7cd002fe40 2024-01-21T08:05:11.894 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.892+0000 7f7ccbfff700 1 -- 172.21.15.46:0/3844944304 <== mgr.14144 v2:172.21.15.46:6800/1810147681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f7ce41026e0 con 0x7f7cd002fe40 2024-01-21T08:05:11.894 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-21T08:05:11.897 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f7cd002fe40 msgr2=0x7f7cd0032300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:11.897 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f7cd002fe40 0x7f7cd0032300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7cd40096a0 tx=0x7f7cd4006b40).stop 2024-01-21T08:05:11.897 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 msgr2=0x7f7ce410cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:11.897 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce410cc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7cd800e040 tx=0x7f7cd8006ee0).stop 2024-01-21T08:05:11.897 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 shutdown_connections 2024-01-21T08:05:11.897 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f7cd002fe40 0x7f7cd0032300 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:11.898 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 --2- 172.21.15.46:0/3844944304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce40fab90 0x7f7ce410cc10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:11.898 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 >> 172.21.15.46:0/3844944304 conn(0x7f7ce40f63b0 msgr2=0x7f7ce40ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:11.898 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 shutdown_connections 2024-01-21T08:05:11.898 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:11.894+0000 7f7cea106700 1 -- 172.21.15.46:0/3844944304 wait complete. 2024-01-21T08:05:12.226 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-21T08:05:12.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:11.001481+0000 mgr.smithi046.lmxnwc (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:12.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:11.002132+0000 mgr.smithi046.lmxnwc (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-21T08:05:12.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:11.893422+0000 mon.smithi046 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.46:0/2859155502' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:12.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.583+0000 7f8ee4087700 1 Processor -- start 2024-01-21T08:05:12.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee4087700 1 -- start start 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee4087700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee4087700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0fb490 con 0x7f8edc0fab80 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee1e23700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee1e23700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43618/0 (socket says 172.21.15.46:43618) 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee1e23700 1 -- 172.21.15.46:0/2016040834 learned_addr learned my addr 172.21.15.46:0/2016040834 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee1e23700 1 -- 172.21.15.46:0/2016040834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc0fb5d0 con 0x7f8edc0fab80 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.584+0000 7f8ee1e23700 1 --2- 172.21.15.46:0/2016040834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc0fafa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8ecc00b870 tx=0x7f8ecc006f70).ready entity=mon.0 client_cookie=69456fb7f63f276a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee0e21700 1 -- 172.21.15.46:0/2016040834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ecc00da20 con 0x7f8edc0fab80 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee0e21700 1 -- 172.21.15.46:0/2016040834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8ecc010470 con 0x7f8edc0fab80 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee0e21700 1 -- 172.21.15.46:0/2016040834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ecc00f5b0 con 0x7f8edc0fab80 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee4087700 1 -- 172.21.15.46:0/2016040834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 msgr2=0x7f8edc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee4087700 1 --2- 172.21.15.46:0/2016040834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc0fafa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8ecc00b870 tx=0x7f8ecc006f70).stop 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee4087700 1 -- 172.21.15.46:0/2016040834 shutdown_connections 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee4087700 1 --2- 172.21.15.46:0/2016040834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc0fafa0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee4087700 1 -- 172.21.15.46:0/2016040834 >> 172.21.15.46:0/2016040834 conn(0x7f8edc0f63c0 msgr2=0x7f8edc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee4087700 1 -- 172.21.15.46:0/2016040834 shutdown_connections 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.585+0000 7f8ee4087700 1 -- 172.21.15.46:0/2016040834 wait complete. 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.586+0000 7f8ee4087700 1 Processor -- start 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.586+0000 7f8ee4087700 1 -- start start 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.586+0000 7f8ee4087700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc108750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.586+0000 7f8ee4087700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc108c40 con 0x7f8edc0fab80 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.586+0000 7f8ee1e23700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.586+0000 7f8ee1e23700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43632/0 (socket says 172.21.15.46:43632) 2024-01-21T08:05:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.586+0000 7f8ee1e23700 1 -- 172.21.15.46:0/2956784941 learned_addr learned my addr 172.21.15.46:0/2956784941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.587+0000 7f8ee1e23700 1 -- 172.21.15.46:0/2956784941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc00b480 con 0x7f8edc0fab80 2024-01-21T08:05:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.587+0000 7f8ee1e23700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc108750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8ecc006a40 tx=0x7f8ecc006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.587+0000 7f8eca7fc700 1 -- 172.21.15.46:0/2956784941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ecc00f3d0 con 0x7f8edc0fab80 2024-01-21T08:05:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.587+0000 7f8eca7fc700 1 -- 172.21.15.46:0/2956784941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8ecc00fef0 con 0x7f8edc0fab80 2024-01-21T08:05:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.587+0000 7f8eca7fc700 1 -- 172.21.15.46:0/2956784941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ecc019d40 con 0x7f8edc0fab80 2024-01-21T08:05:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.587+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8edc108e40 con 0x7f8edc0fab80 2024-01-21T08:05:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.587+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc109250 con 0x7f8edc0fab80 2024-01-21T08:05:12.594 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.588+0000 7f8eca7fc700 1 -- 172.21.15.46:0/2956784941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f8ecc02b060 con 0x7f8edc0fab80 2024-01-21T08:05:12.594 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.588+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f8ec402feb0 0x7f8ec4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:12.594 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.588+0000 7f8eca7fc700 1 -- 172.21.15.46:0/2956784941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8ecc044230 con 0x7f8edc0fab80 2024-01-21T08:05:12.594 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.588+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8edc045ad0 con 0x7f8edc0fab80 2024-01-21T08:05:12.594 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.588+0000 7f8ee1622700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f8ec402feb0 0x7f8ec4032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:12.594 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.589+0000 7f8ee1622700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f8ec402feb0 0x7f8ec4032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8ed80096a0 tx=0x7f8ed8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:12.595 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.593+0000 7f8eca7fc700 1 -- 172.21.15.46:0/2956784941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ecc018ba0 con 0x7f8edc0fab80 2024-01-21T08:05:12.752 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.749+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f8edc1025d0 con 0x7f8edc0fab80 2024-01-21T08:05:12.755 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.752+0000 7f8eca7fc700 1 -- 172.21.15.46:0/2956784941 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8ecc018ba0 con 0x7f8edc0fab80 2024-01-21T08:05:12.757 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f8ec402feb0 msgr2=0x7f8ec4032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f8ec402feb0 0x7f8ec4032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8ed80096a0 tx=0x7f8ed8006b40).stop 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 msgr2=0x7f8edc108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc108750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8ecc006a40 tx=0x7f8ecc006f70).stop 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 shutdown_connections 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f8ec402feb0 0x7f8ec4032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 --2- 172.21.15.46:0/2956784941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8edc0fab80 0x7f8edc108750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.754+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 >> 172.21.15.46:0/2956784941 conn(0x7f8edc0f63c0 msgr2=0x7f8edc0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.755+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 shutdown_connections 2024-01-21T08:05:12.758 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:12.755+0000 7f8ee4087700 1 -- 172.21.15.46:0/2956784941 wait complete. 2024-01-21T08:05:13.108 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525bfff700 1 Processor -- start 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525bfff700 1 -- start start 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c100b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525bfff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0fc580 con 0x7f525c0703d0 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43634/0 (socket says 172.21.15.46:43634) 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525affd700 1 -- 172.21.15.46:0/3416561373 learned_addr learned my addr 172.21.15.46:0/3416561373 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.457+0000 7f525affd700 1 -- 172.21.15.46:0/3416561373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c101080 con 0x7f525c0703d0 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.458+0000 7f525affd700 1 --2- 172.21.15.46:0/3416561373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c100b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f524400b5f0 tx=0x7f5244006ee0).ready entity=mon.0 client_cookie=ffa71d3e6a4947f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.458+0000 7f5259ffb700 1 -- 172.21.15.46:0/3416561373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f524400d600 con 0x7f525c0703d0 2024-01-21T08:05:13.462 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.458+0000 7f5259ffb700 1 -- 172.21.15.46:0/3416561373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f524400dbd0 con 0x7f525c0703d0 2024-01-21T08:05:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.458+0000 7f5259ffb700 1 -- 172.21.15.46:0/3416561373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5244011c00 con 0x7f525c0703d0 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.458+0000 7f525bfff700 1 -- 172.21.15.46:0/3416561373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 msgr2=0x7f525c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.458+0000 7f525bfff700 1 --2- 172.21.15.46:0/3416561373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c100b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f524400b5f0 tx=0x7f5244006ee0).stop 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.459+0000 7f525bfff700 1 -- 172.21.15.46:0/3416561373 shutdown_connections 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.459+0000 7f525bfff700 1 --2- 172.21.15.46:0/3416561373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c100b90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.459+0000 7f525bfff700 1 -- 172.21.15.46:0/3416561373 >> 172.21.15.46:0/3416561373 conn(0x7f525c0f6210 msgr2=0x7f525c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.459+0000 7f525bfff700 1 -- 172.21.15.46:0/3416561373 shutdown_connections 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.459+0000 7f525bfff700 1 -- 172.21.15.46:0/3416561373 wait complete. 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.459+0000 7f525bfff700 1 Processor -- start 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525bfff700 1 -- start start 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c10ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525bfff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c10cf60 con 0x7f525c0703d0 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c10ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c10ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43648/0 (socket says 172.21.15.46:43648) 2024-01-21T08:05:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525affd700 1 -- 172.21.15.46:0/3615496000 learned_addr learned my addr 172.21.15.46:0/3615496000 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525affd700 1 -- 172.21.15.46:0/3615496000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524400b200 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f525affd700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c10ca70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f524400b190 tx=0x7f5244006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f52537fe700 1 -- 172.21.15.46:0/3615496000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f524401b410 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.460+0000 7f52537fe700 1 -- 172.21.15.46:0/3615496000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f524401b9e0 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.461+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c10d160 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.461+0000 7f52537fe700 1 -- 172.21.15.46:0/3615496000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f524401a880 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.461+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c10d570 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.461+0000 7f52537fe700 1 -- 172.21.15.46:0/3615496000 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f524400f070 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.462+0000 7f52537fe700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f523c02feb0 0x7f523c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.462+0000 7f52537fe700 1 -- 172.21.15.46:0/3615496000 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f52440413b0 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.462+0000 7f525a7fc700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f523c02feb0 0x7f523c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.462+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c059100 con 0x7f525c0703d0 2024-01-21T08:05:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.462+0000 7f525a7fc700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f523c02feb0 0x7f523c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f524c0096a0 tx=0x7f524c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:13.469 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.466+0000 7f52537fe700 1 -- 172.21.15.46:0/3615496000 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5244016070 con 0x7f525c0703d0 2024-01-21T08:05:13.624 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.622+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f525c045ad0 con 0x7f525c0703d0 2024-01-21T08:05:13.627 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.624+0000 7f52537fe700 1 -- 172.21.15.46:0/3615496000 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f524401a9e0 con 0x7f525c0703d0 2024-01-21T08:05:13.629 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f523c02feb0 msgr2=0x7f523c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:13.630 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f523c02feb0 0x7f523c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f524c0096a0 tx=0x7f524c006b40).stop 2024-01-21T08:05:13.630 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 msgr2=0x7f525c10ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:13.630 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c10ca70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f524400b190 tx=0x7f5244006ee0).stop 2024-01-21T08:05:13.631 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 shutdown_connections 2024-01-21T08:05:13.631 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f523c02feb0 0x7f523c032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:13.631 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 --2- 172.21.15.46:0/3615496000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0703d0 0x7f525c10ca70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:13.631 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 >> 172.21.15.46:0/3615496000 conn(0x7f525c0f6210 msgr2=0x7f525c0f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:13.631 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 shutdown_connections 2024-01-21T08:05:13.631 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:13.627+0000 7f525bfff700 1 -- 172.21.15.46:0/3615496000 wait complete. 2024-01-21T08:05:13.911 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:11.890669+0000 mgr.smithi046.lmxnwc (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:13.911 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:11.891166+0000 mgr.smithi046.lmxnwc (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-21T08:05:13.911 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:12.754172+0000 mon.smithi046 (mon.0) 83 : audit [INF] from='client.? 172.21.15.46:0/2956784941' entity='client.admin' 2024-01-21T08:05:13.911 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:13.626368+0000 mon.smithi046 (mon.0) 84 : audit [INF] from='client.? 172.21.15.46:0/3615496000' entity='client.admin' 2024-01-21T08:05:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:Enabling the dashboard module... 2024-01-21T08:05:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.323+0000 7fd3d7f1c700 1 Processor -- start 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.323+0000 7fd3d7f1c700 1 -- start start 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.323+0000 7fd3d7f1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.323+0000 7fd3d7f1c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00fedd0 con 0x7fd3d0070430 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.323+0000 7fd3d5cb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.323+0000 7fd3d5cb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43650/0 (socket says 172.21.15.46:43650) 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.323+0000 7fd3d5cb8700 1 -- 172.21.15.46:0/4257582167 learned_addr learned my addr 172.21.15.46:0/4257582167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.324+0000 7fd3d5cb8700 1 -- 172.21.15.46:0/4257582167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d00fef10 con 0x7fd3d0070430 2024-01-21T08:05:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.324+0000 7fd3d5cb8700 1 --2- 172.21.15.46:0/4257582167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fe8e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd3c000b5f0 tx=0x7fd3c0006ee0).ready entity=mon.0 client_cookie=1738048fe0875b18 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.324+0000 7fd3d4cb6700 1 -- 172.21.15.46:0/4257582167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c000d600 con 0x7fd3d0070430 2024-01-21T08:05:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.324+0000 7fd3d4cb6700 1 -- 172.21.15.46:0/4257582167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd3c000dbd0 con 0x7fd3d0070430 2024-01-21T08:05:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.324+0000 7fd3d4cb6700 1 -- 172.21.15.46:0/4257582167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c0011c00 con 0x7fd3d0070430 2024-01-21T08:05:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/4257582167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 msgr2=0x7fd3d00fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 --2- 172.21.15.46:0/4257582167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fe8e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd3c000b5f0 tx=0x7fd3c0006ee0).stop 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/4257582167 shutdown_connections 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 --2- 172.21.15.46:0/4257582167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fe8e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/4257582167 >> 172.21.15.46:0/4257582167 conn(0x7fd3d00f63c0 msgr2=0x7fd3d00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/4257582167 shutdown_connections 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/4257582167 wait complete. 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 Processor -- start 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 -- start start 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.325+0000 7fd3d7f1c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00fa7e0 con 0x7fd3d0070430 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3d5cb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3d5cb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43660/0 (socket says 172.21.15.46:43660) 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3d5cb8700 1 -- 172.21.15.46:0/2100620421 learned_addr learned my addr 172.21.15.46:0/2100620421 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3d5cb8700 1 -- 172.21.15.46:0/2100620421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c000b200 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3d5cb8700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fc360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd3c0002f40 tx=0x7fd3c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c0011690 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd3c001b880 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c001a6b0 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d00fa980 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.326+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d00fad90 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.327+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd3c000f070 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.327+0000 7fd3be7fc700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fd3b802feb0 0x7fd3b8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.327+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd3c0041310 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.327+0000 7fd3d54b7700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fd3b802feb0 0x7fd3b8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.327+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d0059100 con 0x7fd3d0070430 2024-01-21T08:05:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.328+0000 7fd3d54b7700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fd3b802feb0 0x7fd3b8032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3cc0096a0 tx=0x7fd3cc006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:14.334 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.332+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3c001a8e0 con 0x7fd3d0070430 2024-01-21T08:05:14.531 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:14.528+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fd3d0045ad0 con 0x7fd3d0070430 2024-01-21T08:05:15.254 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.251+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd3c001b550 con 0x7fd3d0070430 2024-01-21T08:05:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.253+0000 7fd3be7fc700 1 -- 172.21.15.46:0/2100620421 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fd3c0023360 con 0x7fd3d0070430 2024-01-21T08:05:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.255+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fd3b802feb0 msgr2=0x7fd3b8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.255+0000 7fd3d7f1c700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fd3b802feb0 0x7fd3b8032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3cc0096a0 tx=0x7fd3cc006b40).stop 2024-01-21T08:05:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.255+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 msgr2=0x7fd3d00fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.255+0000 7fd3d7f1c700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fc360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd3c0002f40 tx=0x7fd3c0006ee0).stop 2024-01-21T08:05:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.255+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 shutdown_connections 2024-01-21T08:05:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.255+0000 7fd3d7f1c700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7fd3b802feb0 0x7fd3b8032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.255+0000 7fd3d7f1c700 1 --2- 172.21.15.46:0/2100620421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0070430 0x7fd3d00fc360 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.256+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 >> 172.21.15.46:0/2100620421 conn(0x7fd3d00f63c0 msgr2=0x7fd3d00f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.256+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 shutdown_connections 2024-01-21T08:05:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:15.256+0000 7fd3d7f1c700 1 -- 172.21.15.46:0/2100620421 wait complete. 2024-01-21T08:05:15.544 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:14.530786+0000 mon.smithi046 (mon.0) 85 : audit [INF] from='client.? 172.21.15.46:0/2100620421' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-21T08:05:15.817 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T08:05:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.184+0000 7f4db1dbb700 1 Processor -- start 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.184+0000 7f4db1dbb700 1 -- start start 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4db1dbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4db1dbb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0fb490 con 0x7f4dac0fab80 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4dab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4dab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43686/0 (socket says 172.21.15.46:43686) 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4dab7fe700 1 -- 172.21.15.46:0/3033573483 learned_addr learned my addr 172.21.15.46:0/3033573483 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4dab7fe700 1 -- 172.21.15.46:0/3033573483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0fb5d0 con 0x7f4dac0fab80 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4dab7fe700 1 --2- 172.21.15.46:0/3033573483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac0fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f4d9c00b870 tx=0x7f4d9c006f70).ready entity=mon.0 client_cookie=e773894206a632a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.185+0000 7f4daa7fc700 1 -- 172.21.15.46:0/3033573483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d9c00da20 con 0x7f4dac0fab80 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4daa7fc700 1 -- 172.21.15.46:0/3033573483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d9c010470 con 0x7f4dac0fab80 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4daa7fc700 1 -- 172.21.15.46:0/3033573483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d9c00f5b0 con 0x7f4dac0fab80 2024-01-21T08:05:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4db1dbb700 1 -- 172.21.15.46:0/3033573483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 msgr2=0x7f4dac0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4db1dbb700 1 --2- 172.21.15.46:0/3033573483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac0fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f4d9c00b870 tx=0x7f4d9c006f70).stop 2024-01-21T08:05:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4db1dbb700 1 -- 172.21.15.46:0/3033573483 shutdown_connections 2024-01-21T08:05:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4db1dbb700 1 --2- 172.21.15.46:0/3033573483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac0fafa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4db1dbb700 1 -- 172.21.15.46:0/3033573483 >> 172.21.15.46:0/3033573483 conn(0x7f4dac0f63c0 msgr2=0x7f4dac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4db1dbb700 1 -- 172.21.15.46:0/3033573483 shutdown_connections 2024-01-21T08:05:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.186+0000 7f4db1dbb700 1 -- 172.21.15.46:0/3033573483 wait complete. 2024-01-21T08:05:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.187+0000 7f4db1dbb700 1 Processor -- start 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.187+0000 7f4db1dbb700 1 -- start start 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.187+0000 7f4db1dbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.187+0000 7f4db1dbb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac108c60 con 0x7f4dac0fab80 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.187+0000 7f4dab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.187+0000 7f4dab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43690/0 (socket says 172.21.15.46:43690) 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.187+0000 7f4dab7fe700 1 -- 172.21.15.46:0/2781746014 learned_addr learned my addr 172.21.15.46:0/2781746014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.188+0000 7f4dab7fe700 1 -- 172.21.15.46:0/2781746014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c00b480 con 0x7f4dac0fab80 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.188+0000 7f4dab7fe700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac108770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4d9c006800 tx=0x7f4d9c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.188+0000 7f4d93fff700 1 -- 172.21.15.46:0/2781746014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d9c00f820 con 0x7f4dac0fab80 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.188+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac108e60 con 0x7f4dac0fab80 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.188+0000 7f4d93fff700 1 -- 172.21.15.46:0/2781746014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d9c00fdf0 con 0x7f4dac0fab80 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.188+0000 7f4d93fff700 1 -- 172.21.15.46:0/2781746014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d9c019d50 con 0x7f4dac0fab80 2024-01-21T08:05:16.193 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.188+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac109270 con 0x7f4dac0fab80 2024-01-21T08:05:16.194 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.189+0000 7f4d93fff700 1 -- 172.21.15.46:0/2781746014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f4d9c016030 con 0x7f4dac0fab80 2024-01-21T08:05:16.194 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.189+0000 7f4d93fff700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 0x7f4d98032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:16.194 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.189+0000 7f4d93fff700 1 -- 172.21.15.46:0/2781746014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4d9c044580 con 0x7f4dac0fab80 2024-01-21T08:05:16.194 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.189+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac045ad0 con 0x7f4dac0fab80 2024-01-21T08:05:16.194 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.189+0000 7f4daaffd700 1 -- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 msgr2=0x7f4d98032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1810147681 2024-01-21T08:05:16.194 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.189+0000 7f4daaffd700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 0x7f4d98032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T08:05:16.196 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.193+0000 7f4d93fff700 1 -- 172.21.15.46:0/2781746014 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d9c014070 con 0x7f4dac0fab80 2024-01-21T08:05:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.390+0000 7f4daaffd700 1 -- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 msgr2=0x7f4d98032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1810147681 2024-01-21T08:05:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.390+0000 7f4daaffd700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 0x7f4d98032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T08:05:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.399+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f4dac102670 con 0x7f4dac0fab80 2024-01-21T08:05:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.399+0000 7f4d93fff700 1 -- 172.21.15.46:0/2781746014 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f4d9c022780 con 0x7f4dac0fab80 2024-01-21T08:05:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:05:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-21T08:05:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "available": true, 2024-01-21T08:05:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "active_name": "smithi046.lmxnwc", 2024-01-21T08:05:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T08:05:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:05:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 msgr2=0x7f4d98032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 0x7f4d98032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 msgr2=0x7f4dac108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac108770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4d9c006800 tx=0x7f4d9c006f70).stop 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 shutdown_connections 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4d9802feb0 0x7f4d98032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 --2- 172.21.15.46:0/2781746014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dac0fab80 0x7f4dac108770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 >> 172.21.15.46:0/2781746014 conn(0x7f4dac0f63c0 msgr2=0x7f4dac0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 shutdown_connections 2024-01-21T08:05:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:16.401+0000 7f4db1dbb700 1 -- 172.21.15.46:0/2781746014 wait complete. 2024-01-21T08:05:16.574 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:15.254745+0000 mon.smithi046 (mon.0) 86 : audit [INF] from='client.? 172.21.15.46:0/2100620421' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-21T08:05:16.574 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:15.254811+0000 mon.smithi046 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi046.lmxnwc(active, since 7s) 2024-01-21T08:05:16.708 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for the mgr to restart... 2024-01-21T08:05:16.709 INFO:teuthology.orchestra.run.smithi046.stderr:Waiting for mgr epoch 13... 2024-01-21T08:05:16.709 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T08:05:17.078 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.074+0000 7f4bf6df1700 1 Processor -- start 2024-01-21T08:05:17.078 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf6df1700 1 -- start start 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf6df1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf6df1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00fee50 con 0x7f4bf00703b0 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf4b8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf4b8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fe960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43702/0 (socket says 172.21.15.46:43702) 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf4b8d700 1 -- 172.21.15.46:0/2625643315 learned_addr learned my addr 172.21.15.46:0/2625643315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf4b8d700 1 -- 172.21.15.46:0/2625643315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf00fef90 con 0x7f4bf00703b0 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.075+0000 7f4bf4b8d700 1 --2- 172.21.15.46:0/2625643315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fe960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4be400b5f0 tx=0x7f4be4006ee0).ready entity=mon.0 client_cookie=cd551acfe755933c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:17.079 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.076+0000 7f4bef7fe700 1 -- 172.21.15.46:0/2625643315 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4be400d600 con 0x7f4bf00703b0 2024-01-21T08:05:17.080 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.076+0000 7f4bef7fe700 1 -- 172.21.15.46:0/2625643315 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4be400dbd0 con 0x7f4bf00703b0 2024-01-21T08:05:17.080 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.076+0000 7f4bef7fe700 1 -- 172.21.15.46:0/2625643315 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4be4011c00 con 0x7f4bf00703b0 2024-01-21T08:05:17.080 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.076+0000 7f4bf6df1700 1 -- 172.21.15.46:0/2625643315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 msgr2=0x7f4bf00fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:17.080 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.076+0000 7f4bf6df1700 1 --2- 172.21.15.46:0/2625643315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fe960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4be400b5f0 tx=0x7f4be4006ee0).stop 2024-01-21T08:05:17.080 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.077+0000 7f4bf6df1700 1 -- 172.21.15.46:0/2625643315 shutdown_connections 2024-01-21T08:05:17.080 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.077+0000 7f4bf6df1700 1 --2- 172.21.15.46:0/2625643315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fe960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:17.080 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.077+0000 7f4bf6df1700 1 -- 172.21.15.46:0/2625643315 >> 172.21.15.46:0/2625643315 conn(0x7f4bf00f63c0 msgr2=0x7f4bf00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.077+0000 7f4bf6df1700 1 -- 172.21.15.46:0/2625643315 shutdown_connections 2024-01-21T08:05:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.077+0000 7f4bf6df1700 1 -- 172.21.15.46:0/2625643315 wait complete. 2024-01-21T08:05:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.077+0000 7f4bf6df1700 1 Processor -- start 2024-01-21T08:05:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.077+0000 7f4bf6df1700 1 -- start start 2024-01-21T08:05:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bf6df1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bf6df1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00fa860 con 0x7f4bf00703b0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bf4b8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bf4b8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43712/0 (socket says 172.21.15.46:43712) 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bf4b8d700 1 -- 172.21.15.46:0/4243893428 learned_addr learned my addr 172.21.15.46:0/4243893428 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bf4b8d700 1 -- 172.21.15.46:0/4243893428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be400b200 con 0x7f4bf00703b0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bf4b8d700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fc3e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f4be400b190 tx=0x7f4be4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.078+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4be401b410 con 0x7f4bf00703b0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.079+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4be401b9e0 con 0x7f4bf00703b0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.079+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf00faa00 con 0x7f4bf00703b0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.079+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4be401a8f0 con 0x7f4bf00703b0 2024-01-21T08:05:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.079+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf00fae10 con 0x7f4bf00703b0 2024-01-21T08:05:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.080+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f4be400f070 con 0x7f4bf00703b0 2024-01-21T08:05:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.080+0000 7f4bed7fa700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.080+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4be40414e0 con 0x7f4bf00703b0 2024-01-21T08:05:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.080+0000 7f4beffff700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 msgr2=0x7f4bd4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1810147681 2024-01-21T08:05:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.080+0000 7f4beffff700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T08:05:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.080+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 --> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bf0059100 con 0x7f4bd402feb0 2024-01-21T08:05:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.281+0000 7f4beffff700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 msgr2=0x7f4bd4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1810147681 2024-01-21T08:05:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.281+0000 7f4beffff700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T08:05:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.682+0000 7f4beffff700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 msgr2=0x7f4bd4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1810147681 2024-01-21T08:05:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:17.682+0000 7f4beffff700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T08:05:17.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:16.401340+0000 mon.smithi046 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.46:0/2781746014' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T08:05:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:18.483+0000 7f4beffff700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 msgr2=0x7f4bd4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1810147681 2024-01-21T08:05:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:18.483+0000 7f4beffff700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T08:05:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:20.085+0000 7f4beffff700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 msgr2=0x7f4bd4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1810147681 2024-01-21T08:05:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:20.085+0000 7f4beffff700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T08:05:20.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:20.587+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f4be4025980 con 0x7f4bf00703b0 2024-01-21T08:05:20.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:20.587+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 msgr2=0x7f4bd4032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T08:05:20.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:20.587+0000 7f4bed7fa700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:05:21.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.588+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f4be4022310 con 0x7f4bf00703b0 2024-01-21T08:05:21.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.588+0000 7f4bed7fa700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4bd4033990 0x7f4bd4035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:21.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.588+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bf0059100 con 0x7f4bd4033990 2024-01-21T08:05:21.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.588+0000 7f4beffff700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4bd4033990 0x7f4bd4035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:21.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.589+0000 7f4beffff700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4bd4033990 0x7f4bd4035d80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f4bdc006fd0 tx=0x7f4bdc006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.590+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f4bf0059100 con 0x7f4bd4033990 2024-01-21T08:05:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4bf0045ad0 con 0x7f4bd4033990 2024-01-21T08:05:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bed7fa700 1 -- 172.21.15.46:0/4243893428 <== mgr.14164 v2:172.21.15.46:6800/559524926 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4bf0045ad0 con 0x7f4bd4033990 2024-01-21T08:05:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: { 2024-01-21T08:05:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-21T08:05:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T08:05:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: } 2024-01-21T08:05:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4bd4033990 msgr2=0x7f4bd4035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4bd4033990 0x7f4bd4035d80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f4bdc006fd0 tx=0x7f4bdc006cf0).stop 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 msgr2=0x7f4bf00fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fc3e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f4be400b190 tx=0x7f4be4006ee0).stop 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 shutdown_connections 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4bd4033990 0x7f4bd4035d80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf00703b0 0x7f4bf00fc3e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 --2- 172.21.15.46:0/4243893428 >> [v2:172.21.15.46:6800/1810147681,v1:172.21.15.46:6801/1810147681] conn(0x7f4bd402feb0 0x7f4bd4032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 >> 172.21.15.46:0/4243893428 conn(0x7f4bf00f63c0 msgr2=0x7f4bf00f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.596+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 shutdown_connections 2024-01-21T08:05:21.602 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:21.597+0000 7f4bf6df1700 1 -- 172.21.15.46:0/4243893428 wait complete. 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:20.535525+0000 mon.smithi046 (mon.0) 89 : cluster [INF] Active manager daemon smithi046.lmxnwc restarted 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:20.536048+0000 mon.smithi046 (mon.0) 90 : cluster [INF] Activating manager daemon smithi046.lmxnwc 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:20.537623+0000 mon.smithi046 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:20.588493+0000 mon.smithi046 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi046.lmxnwc(active, starting, since 0.0525557s) 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.589267+0000 mon.smithi046 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.589907+0000 mon.smithi046 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr metadata", "who": "smithi046.lmxnwc", "id": "smithi046.lmxnwc"}]: dispatch 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.590555+0000 mon.smithi046 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T08:05:21.894 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.590732+0000 mon.smithi046 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T08:05:21.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.590896+0000 mon.smithi046 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T08:05:21.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:20.605218+0000 mon.smithi046 (mon.0) 98 : cluster [INF] Manager daemon smithi046.lmxnwc is now available 2024-01-21T08:05:21.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.710834+0000 mon.smithi046 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:05:21.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.712653+0000 mon.smithi046 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:05:21.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.737219+0000 mon.smithi046 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:05:21.895 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:20.741447+0000 mon.smithi046 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:05:21.941 INFO:teuthology.orchestra.run.smithi046.stderr:mgr epoch 13 is available 2024-01-21T08:05:21.941 INFO:teuthology.orchestra.run.smithi046.stderr:Generating a dashboard self-signed certificate... 2024-01-21T08:05:21.942 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-21T08:05:22.312 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.309+0000 7f1ef7236700 1 Processor -- start 2024-01-21T08:05:22.312 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.309+0000 7f1ef7236700 1 -- start start 2024-01-21T08:05:22.312 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.309+0000 7f1ef7236700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef0100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:22.312 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.309+0000 7f1ef7236700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef00fc770 con 0x7f1ef0070390 2024-01-21T08:05:22.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1ef4fd2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef0100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:22.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1ef4fd2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef0100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36722/0 (socket says 172.21.15.46:36722) 2024-01-21T08:05:22.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1ef4fd2700 1 -- 172.21.15.46:0/2736053385 learned_addr learned my addr 172.21.15.46:0/2736053385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:22.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1ef4fd2700 1 -- 172.21.15.46:0/2736053385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef0101270 con 0x7f1ef0070390 2024-01-21T08:05:22.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1ef4fd2700 1 --2- 172.21.15.46:0/2736053385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef0100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1ee400b5f0 tx=0x7f1ee4006ee0).ready entity=mon.0 client_cookie=b55a9d394aa8a656 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:22.314 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1eef7fe700 1 -- 172.21.15.46:0/2736053385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee400d600 con 0x7f1ef0070390 2024-01-21T08:05:22.314 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1eef7fe700 1 -- 172.21.15.46:0/2736053385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ee400dbd0 con 0x7f1ef0070390 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.310+0000 7f1eef7fe700 1 -- 172.21.15.46:0/2736053385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee4011c00 con 0x7f1ef0070390 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 -- 172.21.15.46:0/2736053385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 msgr2=0x7f1ef0100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 --2- 172.21.15.46:0/2736053385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef0100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1ee400b5f0 tx=0x7f1ee4006ee0).stop 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 -- 172.21.15.46:0/2736053385 shutdown_connections 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 --2- 172.21.15.46:0/2736053385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef0100d80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 -- 172.21.15.46:0/2736053385 >> 172.21.15.46:0/2736053385 conn(0x7f1ef00f63c0 msgr2=0x7f1ef00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 -- 172.21.15.46:0/2736053385 shutdown_connections 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 -- 172.21.15.46:0/2736053385 wait complete. 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 Processor -- start 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.311+0000 7f1ef7236700 1 -- start start 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1ef7236700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef010cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1ef7236700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef010d110 con 0x7f1ef0070390 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1ef4fd2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef010cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:22.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1ef4fd2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef010cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36734/0 (socket says 172.21.15.46:36734) 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1ef4fd2700 1 -- 172.21.15.46:0/2537587932 learned_addr learned my addr 172.21.15.46:0/2537587932 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1ef4fd2700 1 -- 172.21.15.46:0/2537587932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee400b200 con 0x7f1ef0070390 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1ef4fd2700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef010cc20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1ee400b190 tx=0x7f1ee4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.312+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee401b410 con 0x7f1ef0070390 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.313+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ee401b9e0 con 0x7f1ef0070390 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.313+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee401a880 con 0x7f1ef0070390 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.313+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef010d310 con 0x7f1ef0070390 2024-01-21T08:05:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.313+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef010d720 con 0x7f1ef0070390 2024-01-21T08:05:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.313+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f1ee400f070 con 0x7f1ef0070390 2024-01-21T08:05:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.313+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef0045ad0 con 0x7f1ef0070390 2024-01-21T08:05:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.314+0000 7f1eed7fa700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ed402fd50 0x7f1ed4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.314+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ee40414d0 con 0x7f1ef0070390 2024-01-21T08:05:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.314+0000 7f1eeffff700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ed402fd50 0x7f1ed4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.314+0000 7f1eeffff700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ed402fd50 0x7f1ed4032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1edc0096a0 tx=0x7f1edc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:22.319 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.318+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ee4022780 con 0x7f1ef0070390 2024-01-21T08:05:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.482+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ef010df90 con 0x7f1ed402fd50 2024-01-21T08:05:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.591+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1ee4025980 con 0x7f1ef0070390 2024-01-21T08:05:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.595+0000 7f1eed7fa700 1 -- 172.21.15.46:0/2537587932 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1ef010df90 con 0x7f1ed402fd50 2024-01-21T08:05:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-21T08:05:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ed402fd50 msgr2=0x7f1ed4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ed402fd50 0x7f1ed4032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1edc0096a0 tx=0x7f1edc006b40).stop 2024-01-21T08:05:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 msgr2=0x7f1ef010cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef010cc20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1ee400b190 tx=0x7f1ee4006ee0).stop 2024-01-21T08:05:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 shutdown_connections 2024-01-21T08:05:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ed402fd50 0x7f1ed4032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 --2- 172.21.15.46:0/2537587932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef0070390 0x7f1ef010cc20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 >> 172.21.15.46:0/2537587932 conn(0x7f1ef00f63c0 msgr2=0x7f1ef00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 shutdown_connections 2024-01-21T08:05:22.601 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:22.597+0000 7f1ef7236700 1 -- 172.21.15.46:0/2537587932 wait complete. 2024-01-21T08:05:22.934 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:21.590383+0000 mgr.smithi046.lmxnwc (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T08:05:22.935 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:21.590490+0000 mon.smithi046 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi046.lmxnwc(active, since 1.05456s) 2024-01-21T08:05:22.935 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:21.596905+0000 mgr.smithi046.lmxnwc (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T08:05:22.973 INFO:teuthology.orchestra.run.smithi046.stderr:Creating initial admin user... 2024-01-21T08:05:22.974 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpq5c7wfp_:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-21T08:05:23.309 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f951c84a700 1 Processor -- start 2024-01-21T08:05:23.309 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f951c84a700 1 -- start start 2024-01-21T08:05:23.309 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f951c84a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f95180fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:23.309 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f951c84a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95180fec20 con 0x7f9518070430 2024-01-21T08:05:23.309 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f95177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f95180fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:23.309 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f95177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f95180fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36750/0 (socket says 172.21.15.46:36750) 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f95177fe700 1 -- 172.21.15.46:0/1731615523 learned_addr learned my addr 172.21.15.46:0/1731615523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.305+0000 7f95177fe700 1 -- 172.21.15.46:0/1731615523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95180fed60 con 0x7f9518070430 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.306+0000 7f95177fe700 1 --2- 172.21.15.46:0/1731615523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f95180fe730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f950800b5f0 tx=0x7f9508006ee0).ready entity=mon.0 client_cookie=52c408eb3d3c7212 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.306+0000 7f95167fc700 1 -- 172.21.15.46:0/1731615523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950800d600 con 0x7f9518070430 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.306+0000 7f95167fc700 1 -- 172.21.15.46:0/1731615523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f950800dbd0 con 0x7f9518070430 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.306+0000 7f95167fc700 1 -- 172.21.15.46:0/1731615523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9508011c00 con 0x7f9518070430 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.306+0000 7f951c84a700 1 -- 172.21.15.46:0/1731615523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 msgr2=0x7f95180fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.306+0000 7f951c84a700 1 --2- 172.21.15.46:0/1731615523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f95180fe730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f950800b5f0 tx=0x7f9508006ee0).stop 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.307+0000 7f951c84a700 1 -- 172.21.15.46:0/1731615523 shutdown_connections 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.307+0000 7f951c84a700 1 --2- 172.21.15.46:0/1731615523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f95180fe730 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.307+0000 7f951c84a700 1 -- 172.21.15.46:0/1731615523 >> 172.21.15.46:0/1731615523 conn(0x7f95180f6210 msgr2=0x7f95180f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.307+0000 7f951c84a700 1 -- 172.21.15.46:0/1731615523 shutdown_connections 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.307+0000 7f951c84a700 1 -- 172.21.15.46:0/1731615523 wait complete. 2024-01-21T08:05:23.310 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.307+0000 7f951c84a700 1 Processor -- start 2024-01-21T08:05:23.311 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.307+0000 7f951c84a700 1 -- start start 2024-01-21T08:05:23.311 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f951c84a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f951810a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:23.311 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f951c84a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951810ad40 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f95177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f951810a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f95177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f951810a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36756/0 (socket says 172.21.15.46:36756) 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f95177fe700 1 -- 172.21.15.46:0/3427086389 learned_addr learned my addr 172.21.15.46:0/3427086389 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f95177fe700 1 -- 172.21.15.46:0/3427086389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950800b200 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f95177fe700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f951810a850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f950800b190 tx=0x7f9508006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f94fffff700 1 -- 172.21.15.46:0/3427086389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950801b410 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f94fffff700 1 -- 172.21.15.46:0/3427086389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f950801b9e0 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951810af40 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f94fffff700 1 -- 172.21.15.46:0/3427086389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950801a880 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.308+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951810b350 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.309+0000 7f94fffff700 1 -- 172.21.15.46:0/3427086389 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f950800f070 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.309+0000 7f94fffff700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f950402feb0 0x7f9504032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.309+0000 7f94fffff700 1 -- 172.21.15.46:0/3427086389 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9508041600 con 0x7f9518070430 2024-01-21T08:05:23.313 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.309+0000 7f9516ffd700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f950402feb0 0x7f9504032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:23.314 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.310+0000 7f9516ffd700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f950402feb0 0x7f9504032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f950c0096a0 tx=0x7f950c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:23.314 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.310+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9518059100 con 0x7f9518070430 2024-01-21T08:05:23.315 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.314+0000 7f94fffff700 1 -- 172.21.15.46:0/3427086389 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9508016070 con 0x7f9518070430 2024-01-21T08:05:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.489+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9518045ad0 con 0x7f950402feb0 2024-01-21T08:05:23.736 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.734+0000 7f94fffff700 1 -- 172.21.15.46:0/3427086389 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9518045ad0 con 0x7f950402feb0 2024-01-21T08:05:23.737 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$WOQd1T6euuCzSY9kdB9G.e/xknIGD2oUPoNHVAZbAbYULksCNhoom", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705824323, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-21T08:05:23.738 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f950402feb0 msgr2=0x7f9504032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f950402feb0 0x7f9504032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f950c0096a0 tx=0x7f950c006b40).stop 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 msgr2=0x7f951810a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f951810a850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f950800b190 tx=0x7f9508006ee0).stop 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 shutdown_connections 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f950402feb0 0x7f9504032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 --2- 172.21.15.46:0/3427086389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9518070430 0x7f951810a850 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 >> 172.21.15.46:0/3427086389 conn(0x7f95180f6210 msgr2=0x7f95180f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 shutdown_connections 2024-01-21T08:05:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:23.736+0000 7f951c84a700 1 -- 172.21.15.46:0/3427086389 wait complete. 2024-01-21T08:05:23.906 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:22.483668+0000 mgr.smithi046.lmxnwc (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:23.907 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:22.592606+0000 mon.smithi046 (mon.0) 104 : cluster [DBG] mgrmap e16: smithi046.lmxnwc(active, since 2s) 2024-01-21T08:05:23.907 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:22.594129+0000 mon.smithi046 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:23.907 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:22.595921+0000 mon.smithi046 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:Fetching dashboard port number... 2024-01-21T08:05:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmprdkwbo5u:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmplmcl2b12:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-21T08:05:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.419+0000 7f0fd3eae700 1 Processor -- start 2024-01-21T08:05:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.419+0000 7f0fd3eae700 1 -- start start 2024-01-21T08:05:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd3eae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd3eae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0fc5c0 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd1c4a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd1c4a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36772/0 (socket says 172.21.15.46:36772) 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd1c4a700 1 -- 172.21.15.46:0/2116726489 learned_addr learned my addr 172.21.15.46:0/2116726489 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd1c4a700 1 -- 172.21.15.46:0/2116726489 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc0fc700 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd1c4a700 1 --2- 172.21.15.46:0/2116726489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0fc0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f0fbc009960 tx=0x7f0fbc0092d0).ready entity=mon.0 client_cookie=f3e948e961c25c53 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd0c48700 1 -- 172.21.15.46:0/2116726489 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fbc00b9c0 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd0c48700 1 -- 172.21.15.46:0/2116726489 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0fbc011400 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.420+0000 7f0fd0c48700 1 -- 172.21.15.46:0/2116726489 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fbc0103a0 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.421+0000 7f0fd3eae700 1 -- 172.21.15.46:0/2116726489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 msgr2=0x7f0fcc0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.421+0000 7f0fd3eae700 1 --2- 172.21.15.46:0/2116726489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0fc0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f0fbc009960 tx=0x7f0fbc0092d0).stop 2024-01-21T08:05:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.421+0000 7f0fd3eae700 1 -- 172.21.15.46:0/2116726489 shutdown_connections 2024-01-21T08:05:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.421+0000 7f0fd3eae700 1 --2- 172.21.15.46:0/2116726489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0fc0d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.421+0000 7f0fd3eae700 1 -- 172.21.15.46:0/2116726489 >> 172.21.15.46:0/2116726489 conn(0x7f0fcc0f74b0 msgr2=0x7f0fcc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.421+0000 7f0fd3eae700 1 -- 172.21.15.46:0/2116726489 shutdown_connections 2024-01-21T08:05:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.421+0000 7f0fd3eae700 1 -- 172.21.15.46:0/2116726489 wait complete. 2024-01-21T08:05:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd3eae700 1 Processor -- start 2024-01-21T08:05:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd3eae700 1 -- start start 2024-01-21T08:05:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd3eae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd3eae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc070a60 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd1c4a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:24.432 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd1c4a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36776/0 (socket says 172.21.15.46:36776) 2024-01-21T08:05:24.432 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd1c4a700 1 -- 172.21.15.46:0/924210824 learned_addr learned my addr 172.21.15.46:0/924210824 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:24.432 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd1c4a700 1 -- 172.21.15.46:0/924210824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc009570 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.432 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fd1c4a700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0725e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0fbc000f80 tx=0x7f0fbc0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:24.432 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.422+0000 7f0fca7fc700 1 -- 172.21.15.46:0/924210824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fbc010820 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.432 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.423+0000 7f0fca7fc700 1 -- 172.21.15.46:0/924210824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0fbc010df0 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.432 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.423+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc070c00 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.423+0000 7f0fca7fc700 1 -- 172.21.15.46:0/924210824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fbc023410 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.423+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc071010 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.423+0000 7f0fca7fc700 1 -- 172.21.15.46:0/924210824 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0fbc00f070 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.424+0000 7f0fca7fc700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0fb802ff00 0x7f0fb80323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.424+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc045ad0 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.424+0000 7f0fca7fc700 1 -- 172.21.15.46:0/924210824 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fbc042140 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.424+0000 7f0fd1449700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0fb802ff00 0x7f0fb80323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.424+0000 7f0fd1449700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0fb802ff00 0x7f0fb80323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0fc00096a0 tx=0x7f0fc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.428+0000 7f0fca7fc700 1 -- 172.21.15.46:0/924210824 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fbc01e020 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.589 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.587+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0fcc0fd9c0 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.587+0000 7f0fca7fc700 1 -- 172.21.15.46:0/924210824 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0fbc0218f0 con 0x7f0fcc0fbcb0 2024-01-21T08:05:24.590 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 8443 2024-01-21T08:05:24.591 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0fb802ff00 msgr2=0x7f0fb80323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0fb802ff00 0x7f0fb80323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0fc00096a0 tx=0x7f0fc0006b40).stop 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 msgr2=0x7f0fcc0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0725e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0fbc000f80 tx=0x7f0fbc0092d0).stop 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 shutdown_connections 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0fb802ff00 0x7f0fb80323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 --2- 172.21.15.46:0/924210824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fcc0fbcb0 0x7f0fcc0725e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 >> 172.21.15.46:0/924210824 conn(0x7f0fcc0f74b0 msgr2=0x7f0fcc0f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 shutdown_connections 2024-01-21T08:05:24.592 INFO:teuthology.orchestra.run.smithi046.stderr:/usr/bin/ceph: 2024-01-21T08:05:24.589+0000 7f0fd3eae700 1 -- 172.21.15.46:0/924210824 wait complete. 2024-01-21T08:05:24.909 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:23.490750+0000 mgr.smithi046.lmxnwc (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:24.909 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:23.734894+0000 mon.smithi046 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:24.910 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:24.588077+0000 mon.smithi046 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.46:0/924210824' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-21T08:05:24.957 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T08:05:24.966 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: disabled 2024-01-21T08:05:24.966 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T08:05:24.973 INFO:teuthology.orchestra.run.smithi046.stderr:systemctl: inactive 2024-01-21T08:05:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:firewalld.service is not enabled 2024-01-21T08:05:24.974 INFO:teuthology.orchestra.run.smithi046.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-21T08:05:24.977 INFO:teuthology.orchestra.run.smithi046.stderr:Ceph Dashboard is now available at: 2024-01-21T08:05:24.977 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.977 INFO:teuthology.orchestra.run.smithi046.stderr: URL: https://smithi046.front.sepia.ceph.com:8443/ 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: User: admin 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: Password: 0inqgv003p 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:You can access the Ceph CLI with: 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: ceph telemetry on 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:For more information see: 2024-01-21T08:05:24.979 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.979 INFO:teuthology.orchestra.run.smithi046.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-21T08:05:24.979 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:05:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:Bootstrap complete. 2024-01-21T08:05:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:Releasing lock 140495455857128 on /run/cephadm/a17ce476-b833-11ee-95b0-87774f69a715.lock 2024-01-21T08:05:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:Lock 140495455857128 released on /run/cephadm/a17ce476-b833-11ee-95b0-87774f69a715.lock 2024-01-21T08:05:25.011 INFO:tasks.cephadm:Fetching config... 2024-01-21T08:05:25.012 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:05:25.012 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-21T08:05:25.070 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-21T08:05:25.070 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:05:25.070 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-21T08:05:25.128 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-21T08:05:25.129 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:05:25.129 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/keyring of=/dev/stdout 2024-01-21T08:05:25.197 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-21T08:05:25.197 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:05:25.198 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-21T08:05:25.256 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-21T08:05:25.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUZNjkANzwqvxqxiv4P3kOpOtqccmeBY2Nw/61ji5JLjt6Rfv5fM4UcykVC5/8YGE5YzleGduKRZYwo1/J/jQWrsEKOKnWUdnRneP4rB4Yc7fRk8oZrBj6i9hfb7yoKFKKLTDTjfW7qM3bumW0LaB5OVSHbu4EFyvsNPbWGtP6Qs1vmyMD4zXNBcVajv61g8bWecX4odTVQtlAvK6sP/9QG3c1mB7TU2FIQSys58enw6RA+cG6JJC7vTdXxkLSj/tnUYpvFuAwU3b4DHJBWfolog+SE2oh4Ya0T7j6GPVd6w/XYzzzxv90FEhc9unFp06lkfAbATVadlMp4d6IJbWutSfsZt3/5ZHmVKcsM8Lu0dAJiD7mGcXQ8ApAneEjrWJ1YUVvRY922h9PVZZO3NuREtl/8yeWFteG5i+CniRQ1rJxUFLjkt9sFMdjBg1g76+Djd4bnhkpoG9frR+6UpxhXbgtRHd8AIPZq6RIGReuEo6jYtylft1N/Ao4UChgxOU= ceph-a17ce476-b833-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T08:05:25.333 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUZNjkANzwqvxqxiv4P3kOpOtqccmeBY2Nw/61ji5JLjt6Rfv5fM4UcykVC5/8YGE5YzleGduKRZYwo1/J/jQWrsEKOKnWUdnRneP4rB4Yc7fRk8oZrBj6i9hfb7yoKFKKLTDTjfW7qM3bumW0LaB5OVSHbu4EFyvsNPbWGtP6Qs1vmyMD4zXNBcVajv61g8bWecX4odTVQtlAvK6sP/9QG3c1mB7TU2FIQSys58enw6RA+cG6JJC7vTdXxkLSj/tnUYpvFuAwU3b4DHJBWfolog+SE2oh4Ya0T7j6GPVd6w/XYzzzxv90FEhc9unFp06lkfAbATVadlMp4d6IJbWutSfsZt3/5ZHmVKcsM8Lu0dAJiD7mGcXQ8ApAneEjrWJ1YUVvRY922h9PVZZO3NuREtl/8yeWFteG5i+CniRQ1rJxUFLjkt9sFMdjBg1g76+Djd4bnhkpoG9frR+6UpxhXbgtRHd8AIPZq6RIGReuEo6jYtylft1N/Ao4UChgxOU= ceph-a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:05:25.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUZNjkANzwqvxqxiv4P3kOpOtqccmeBY2Nw/61ji5JLjt6Rfv5fM4UcykVC5/8YGE5YzleGduKRZYwo1/J/jQWrsEKOKnWUdnRneP4rB4Yc7fRk8oZrBj6i9hfb7yoKFKKLTDTjfW7qM3bumW0LaB5OVSHbu4EFyvsNPbWGtP6Qs1vmyMD4zXNBcVajv61g8bWecX4odTVQtlAvK6sP/9QG3c1mB7TU2FIQSys58enw6RA+cG6JJC7vTdXxkLSj/tnUYpvFuAwU3b4DHJBWfolog+SE2oh4Ya0T7j6GPVd6w/XYzzzxv90FEhc9unFp06lkfAbATVadlMp4d6IJbWutSfsZt3/5ZHmVKcsM8Lu0dAJiD7mGcXQ8ApAneEjrWJ1YUVvRY922h9PVZZO3NuREtl/8yeWFteG5i+CniRQ1rJxUFLjkt9sFMdjBg1g76+Djd4bnhkpoG9frR+6UpxhXbgtRHd8AIPZq6RIGReuEo6jYtylft1N/Ao4UChgxOU= ceph-a17ce476-b833-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T08:05:25.391 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUZNjkANzwqvxqxiv4P3kOpOtqccmeBY2Nw/61ji5JLjt6Rfv5fM4UcykVC5/8YGE5YzleGduKRZYwo1/J/jQWrsEKOKnWUdnRneP4rB4Yc7fRk8oZrBj6i9hfb7yoKFKKLTDTjfW7qM3bumW0LaB5OVSHbu4EFyvsNPbWGtP6Qs1vmyMD4zXNBcVajv61g8bWecX4odTVQtlAvK6sP/9QG3c1mB7TU2FIQSys58enw6RA+cG6JJC7vTdXxkLSj/tnUYpvFuAwU3b4DHJBWfolog+SE2oh4Ya0T7j6GPVd6w/XYzzzxv90FEhc9unFp06lkfAbATVadlMp4d6IJbWutSfsZt3/5ZHmVKcsM8Lu0dAJiD7mGcXQ8ApAneEjrWJ1YUVvRY922h9PVZZO3NuREtl/8yeWFteG5i+CniRQ1rJxUFLjkt9sFMdjBg1g76+Djd4bnhkpoG9frR+6UpxhXbgtRHd8AIPZq6RIGReuEo6jYtylft1N/Ao4UChgxOU= ceph-a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:05:25.403 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.547+0000 7f810c819700 1 -- 172.21.15.46:0/982556579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 msgr2=0x7f81080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.547+0000 7f810c819700 1 --2- 172.21.15.46:0/982556579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81080f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f80f8009fb0 tx=0x7f80f8009b30).stop 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 -- 172.21.15.46:0/982556579 shutdown_connections 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 --2- 172.21.15.46:0/982556579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81080f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 -- 172.21.15.46:0/982556579 >> 172.21.15.46:0/982556579 conn(0x7f81080f0b70 msgr2=0x7f81080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 -- 172.21.15.46:0/982556579 shutdown_connections 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 -- 172.21.15.46:0/982556579 wait complete. 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 Processor -- start 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 -- start start 2024-01-21T08:05:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81081055f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810c819700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8108105ae0 con 0x7f81080f5410 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81081055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81081055f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36792/0 (socket says 172.21.15.46:36792) 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.548+0000 7f810659c700 1 -- 172.21.15.46:0/2541178940 learned_addr learned my addr 172.21.15.46:0/2541178940 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.549+0000 7f810659c700 1 -- 172.21.15.46:0/2541178940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f8005040 con 0x7f81080f5410 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.549+0000 7f810659c700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81081055f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f80f8004dd0 tx=0x7f80f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.549+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f8009320 con 0x7f81080f5410 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.549+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f80f8013680 con 0x7f81080f5410 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.549+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8108105ce0 con 0x7f81080f5410 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.549+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f801ecb0 con 0x7f81080f5410 2024-01-21T08:05:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.549+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8108106170 con 0x7f81080f5410 2024-01-21T08:05:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.550+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f80f8014420 con 0x7f81080f5410 2024-01-21T08:05:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.550+0000 7f80f6ffd700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f80f002ff00 0x7f80f00323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.550+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80f8040c70 con 0x7f81080f5410 2024-01-21T08:05:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.550+0000 7f8105d9b700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f80f002ff00 0x7f80f00323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.550+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8108059100 con 0x7f81080f5410 2024-01-21T08:05:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.550+0000 7f8105d9b700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f80f002ff00 0x7f80f00323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f80fc0096a0 tx=0x7f80fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:26.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.554+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80f801c910 con 0x7f81080f5410 2024-01-21T08:05:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.622+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80f80273f0 con 0x7f81080f5410 2024-01-21T08:05:26.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.715+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f8108045ad0 con 0x7f81080f5410 2024-01-21T08:05:26.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.718+0000 7f80f6ffd700 1 -- 172.21.15.46:0/2541178940 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f80f801e420 con 0x7f81080f5410 2024-01-21T08:05:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.719+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f80f002ff00 msgr2=0x7f80f00323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.719+0000 7f810c819700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f80f002ff00 0x7f80f00323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f80fc0096a0 tx=0x7f80fc006b40).stop 2024-01-21T08:05:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.719+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 msgr2=0x7f81081055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.719+0000 7f810c819700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81081055f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f80f8004dd0 tx=0x7f80f800b040).stop 2024-01-21T08:05:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.720+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 shutdown_connections 2024-01-21T08:05:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.720+0000 7f810c819700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f80f002ff00 0x7f80f00323c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.720+0000 7f810c819700 1 --2- 172.21.15.46:0/2541178940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81080f5410 0x7f81081055f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.720+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 >> 172.21.15.46:0/2541178940 conn(0x7f81080f0b70 msgr2=0x7f81080f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.720+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 shutdown_connections 2024-01-21T08:05:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:26.720+0000 7f810c819700 1 -- 172.21.15.46:0/2541178940 wait complete. 2024-01-21T08:05:27.088 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi078 2024-01-21T08:05:27.089 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:05:27.089 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/etc/ceph/ceph.conf 2024-01-21T08:05:27.111 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:05:27.111 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T08:05:27.172 INFO:tasks.cephadm:Adding host smithi078 to orchestrator... 2024-01-21T08:05:27.172 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch host add smithi078 2024-01-21T08:05:27.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:26.624184+0000 mon.smithi046 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi046.lmxnwc(active, since 6s) 2024-01-21T08:05:27.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:26.718815+0000 mon.smithi046 (mon.0) 110 : audit [INF] from='client.? 172.21.15.46:0/2541178940' entity='client.admin' 2024-01-21T08:05:29.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:28.752866+0000 mon.smithi046 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:29.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:28.978693+0000 mon.smithi046 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:29.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:29.719440+0000 mon.smithi046 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:33.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:31.982130+0000 mon.smithi046 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:33.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:32.242298+0000 mon.smithi046 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:33.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:32.244359+0000 mon.smithi046 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:33.241 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:32.250782+0000 mgr.smithi046.lmxnwc (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi046 on smithi046 2024-01-21T08:05:41.882 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:40.591873+0000 mgr.smithi046.lmxnwc (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:44.471 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:42.592110+0000 mgr.smithi046.lmxnwc (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:44.471 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:43.103069+0000 mon.smithi046 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:44.471 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:43.104036+0000 mon.smithi046 (mon.0) 118 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T08:05:44.471 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:43.105599+0000 mon.smithi046 (mon.0) 119 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T08:05:44.471 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:43.106178+0000 mon.smithi046 (mon.0) 120 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:05:44.471 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:43.106717+0000 mgr.smithi046.lmxnwc (mgr.14164) 8 : cephadm [INF] Deploying daemon crash.smithi046 on smithi046 2024-01-21T08:05:45.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:44.592339+0000 mgr.smithi046.lmxnwc (mgr.14164) 9 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:45.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:44.627170+0000 mon.smithi046 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:45.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:44.773058+0000 mon.smithi046 (mon.0) 122 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:45.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:44.774904+0000 mon.smithi046 (mon.0) 123 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:45.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:44.775323+0000 mon.smithi046 (mon.0) 124 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T08:05:45.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:44.775729+0000 mgr.smithi046.lmxnwc (mgr.14164) 10 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T08:05:45.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:44.778102+0000 mon.smithi046 (mon.0) 125 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:45.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:44.781381+0000 mgr.smithi046.lmxnwc (mgr.14164) 11 : cephadm [INF] Deploying daemon grafana.smithi046 on smithi046 2024-01-21T08:05:47.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:46.592818+0000 mgr.smithi046.lmxnwc (mgr.14164) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.258+0000 7f7b98489700 1 -- 172.21.15.78:0/2881494953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 msgr2=0x7f7b900f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.258+0000 7f7b98489700 1 --2- 172.21.15.78:0/2881494953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b900f9190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7b80009fb0 tx=0x7f7b80009b30).stop 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.258+0000 7f7b98489700 1 -- 172.21.15.78:0/2881494953 shutdown_connections 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.258+0000 7f7b98489700 1 --2- 172.21.15.78:0/2881494953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b900f9190 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.258+0000 7f7b98489700 1 -- 172.21.15.78:0/2881494953 >> 172.21.15.78:0/2881494953 conn(0x7f7b900f0b70 msgr2=0x7f7b900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.258+0000 7f7b98489700 1 -- 172.21.15.78:0/2881494953 shutdown_connections 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.258+0000 7f7b98489700 1 -- 172.21.15.78:0/2881494953 wait complete. 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b98489700 1 Processor -- start 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b98489700 1 -- start start 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b98489700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b901066f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b98489700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90106be0 con 0x7f7b900601e0 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b96225700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b901066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b96225700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b901066f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:56032/0 (socket says 172.21.15.78:56032) 2024-01-21T08:05:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b96225700 1 -- 172.21.15.78:0/1833342189 learned_addr learned my addr 172.21.15.78:0/1833342189 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b96225700 1 -- 172.21.15.78:0/1833342189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80005040 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.259+0000 7f7b96225700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b901066f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7b80004dd0 tx=0x7f7b80009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.260+0000 7f7b86ffd700 1 -- 172.21.15.78:0/1833342189 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b80018070 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.260+0000 7f7b86ffd700 1 -- 172.21.15.78:0/1833342189 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b80013410 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.260+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b90106de0 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.260+0000 7f7b86ffd700 1 -- 172.21.15.78:0/1833342189 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b8001c3c0 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.260+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b90107270 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.260+0000 7f7b86ffd700 1 -- 172.21.15.78:0/1833342189 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b8001d420 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.260+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90045ad0 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.261+0000 7f7b86ffd700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7b7c02ff60 0x7f7b7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.261+0000 7f7b86ffd700 1 -- 172.21.15.78:0/1833342189 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b80041510 con 0x7f7b900601e0 2024-01-21T08:05:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.261+0000 7f7b95a24700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7b7c02ff60 0x7f7b7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:48.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.261+0000 7f7b95a24700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7b7c02ff60 0x7f7b7c032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7b8c0096a0 tx=0x7f7b8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.265+0000 7f7b86ffd700 1 -- 172.21.15.78:0/1833342189 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b8001d820 con 0x7f7b900601e0 2024-01-21T08:05:48.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:48.435+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b900f1dc0 con 0x7f7b7c02ff60 2024-01-21T08:05:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.033+0000 7f7b86ffd700 1 -- 172.21.15.78:0/1833342189 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f7b900f1dc0 con 0x7f7b7c02ff60 2024-01-21T08:05:49.037 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi078' 2024-01-21T08:05:49.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7b7c02ff60 msgr2=0x7f7b7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:49.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7b7c02ff60 0x7f7b7c032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7b8c0096a0 tx=0x7f7b8c006b40).stop 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 msgr2=0x7f7b901066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b901066f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7b80004dd0 tx=0x7f7b80009b30).stop 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 shutdown_connections 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7b7c02ff60 0x7f7b7c032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 --2- 172.21.15.78:0/1833342189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900601e0 0x7f7b901066f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 >> 172.21.15.78:0/1833342189 conn(0x7f7b900f0b70 msgr2=0x7f7b900f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 shutdown_connections 2024-01-21T08:05:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:49.035+0000 7f7b98489700 1 -- 172.21.15.78:0/1833342189 wait complete. 2024-01-21T08:05:49.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch host ls --format=json 2024-01-21T08:05:50.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:48.438156+0000 mgr.smithi046.lmxnwc (mgr.14164) 13 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:50.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:48.593227+0000 mgr.smithi046.lmxnwc (mgr.14164) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:50.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:49.034441+0000 mon.smithi046 (mon.0) 126 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:50.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:49.034898+0000 mgr.smithi046.lmxnwc (mgr.14164) 15 : cephadm [INF] Added host smithi078 2024-01-21T08:05:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.585+0000 7ff19d4ec700 1 -- 172.21.15.78:0/310965374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 msgr2=0x7ff1980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.585+0000 7ff19d4ec700 1 --2- 172.21.15.78:0/310965374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff1980f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff180009fb0 tx=0x7ff180009b30).stop 2024-01-21T08:05:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.585+0000 7ff19d4ec700 1 -- 172.21.15.78:0/310965374 shutdown_connections 2024-01-21T08:05:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.585+0000 7ff19d4ec700 1 --2- 172.21.15.78:0/310965374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff1980f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.585+0000 7ff19d4ec700 1 -- 172.21.15.78:0/310965374 >> 172.21.15.78:0/310965374 conn(0x7ff1980f1c70 msgr2=0x7ff1980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.585+0000 7ff19d4ec700 1 -- 172.21.15.78:0/310965374 shutdown_connections 2024-01-21T08:05:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.586+0000 7ff19d4ec700 1 -- 172.21.15.78:0/310965374 wait complete. 2024-01-21T08:05:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.586+0000 7ff19d4ec700 1 Processor -- start 2024-01-21T08:05:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.586+0000 7ff19d4ec700 1 -- start start 2024-01-21T08:05:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.586+0000 7ff19d4ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff198106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.586+0000 7ff19d4ec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff198106c20 con 0x7ff1980f6510 2024-01-21T08:05:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff196ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff198106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff196ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff198106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:56050/0 (socket says 172.21.15.78:56050) 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff196ffd700 1 -- 172.21.15.78:0/3988228657 learned_addr learned my addr 172.21.15.78:0/3988228657 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff196ffd700 1 -- 172.21.15.78:0/3988228657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180005040 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff196ffd700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff198106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff180014000 tx=0x7ff18000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff18f7fe700 1 -- 172.21.15.78:0/3988228657 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1800095f0 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff198106e20 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff18f7fe700 1 -- 172.21.15.78:0/3988228657 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff180007e70 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff18f7fe700 1 -- 172.21.15.78:0/3988228657 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff180014ea0 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.587+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1981072b0 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.588+0000 7ff18f7fe700 1 -- 172.21.15.78:0/3988228657 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1800079c0 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.588+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff198045ad0 con 0x7ff1980f6510 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.588+0000 7ff18f7fe700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff18402ffb0 0x7ff184032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.588+0000 7ff18f7fe700 1 -- 172.21.15.78:0/3988228657 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff180040920 con 0x7ff1980f6510 2024-01-21T08:05:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.588+0000 7ff1967fc700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff18402ffb0 0x7ff184032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.589+0000 7ff1967fc700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff18402ffb0 0x7ff184032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff1880096a0 tx=0x7ff188006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.593+0000 7ff18f7fe700 1 -- 172.21.15.78:0/3988228657 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff18000f950 con 0x7ff1980f6510 2024-01-21T08:05:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.765+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1980f2e70 con 0x7ff18402ffb0 2024-01-21T08:05:50.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff18f7fe700 1 -- 172.21.15.78:0/3988228657 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7ff1980f2e70 con 0x7ff18402ffb0 2024-01-21T08:05:50.770 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:05:50.770 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "smithi078", "hostname": "smithi078", "labels": [], "status": ""}] 2024-01-21T08:05:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff18402ffb0 msgr2=0x7ff184032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff18402ffb0 0x7ff184032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff1880096a0 tx=0x7ff188006b40).stop 2024-01-21T08:05:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 msgr2=0x7ff198106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff198106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff180014000 tx=0x7ff18000b040).stop 2024-01-21T08:05:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 shutdown_connections 2024-01-21T08:05:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff18402ffb0 0x7ff184032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 --2- 172.21.15.78:0/3988228657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1980f6510 0x7ff198106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 >> 172.21.15.78:0/3988228657 conn(0x7ff1980f1c70 msgr2=0x7ff1980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 shutdown_connections 2024-01-21T08:05:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:50.768+0000 7ff19d4ec700 1 -- 172.21.15.78:0/3988228657 wait complete. 2024-01-21T08:05:51.195 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-21T08:05:51.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd crush tunables default 2024-01-21T08:05:52.122 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:50.593498+0000 mgr.smithi046.lmxnwc (mgr.14164) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:52.122 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:50.768292+0000 mgr.smithi046.lmxnwc (mgr.14164) 17 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T08:05:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.304+0000 7fb161e4e700 1 -- 172.21.15.46:0/2357438522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 msgr2=0x7fb15c0f5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.304+0000 7fb161e4e700 1 --2- 172.21.15.46:0/2357438522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c0f5560 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fb150009fb0 tx=0x7fb150009b30).stop 2024-01-21T08:05:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.304+0000 7fb161e4e700 1 -- 172.21.15.46:0/2357438522 shutdown_connections 2024-01-21T08:05:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.304+0000 7fb161e4e700 1 --2- 172.21.15.46:0/2357438522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c0f5560 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.304+0000 7fb161e4e700 1 -- 172.21.15.46:0/2357438522 >> 172.21.15.46:0/2357438522 conn(0x7fb15c0f08c0 msgr2=0x7fb15c0f2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.304+0000 7fb161e4e700 1 -- 172.21.15.46:0/2357438522 shutdown_connections 2024-01-21T08:05:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.304+0000 7fb161e4e700 1 -- 172.21.15.46:0/2357438522 wait complete. 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.305+0000 7fb161e4e700 1 Processor -- start 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.305+0000 7fb161e4e700 1 -- start start 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.305+0000 7fb161e4e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c1086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.305+0000 7fb161e4e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c108bb0 con 0x7fb15c0f5140 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.305+0000 7fb160e4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c1086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.305+0000 7fb160e4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c1086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57232/0 (socket says 172.21.15.46:57232) 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.305+0000 7fb160e4c700 1 -- 172.21.15.46:0/3514814619 learned_addr learned my addr 172.21.15.46:0/3514814619 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.306+0000 7fb160e4c700 1 -- 172.21.15.46:0/3514814619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb150005040 con 0x7fb15c0f5140 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.306+0000 7fb160e4c700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c1086c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb150013510 tx=0x7fb15000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.306+0000 7fb1597fa700 1 -- 172.21.15.46:0/3514814619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb150009320 con 0x7fb15c0f5140 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.306+0000 7fb1597fa700 1 -- 172.21.15.46:0/3514814619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb150013f00 con 0x7fb15c0f5140 2024-01-21T08:05:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.306+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c108db0 con 0x7fb15c0f5140 2024-01-21T08:05:52.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.306+0000 7fb1597fa700 1 -- 172.21.15.46:0/3514814619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb150014d90 con 0x7fb15c0f5140 2024-01-21T08:05:52.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.306+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15c109240 con 0x7fb15c0f5140 2024-01-21T08:05:52.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.307+0000 7fb1597fa700 1 -- 172.21.15.46:0/3514814619 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1500263a0 con 0x7fb15c0f5140 2024-01-21T08:05:52.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.307+0000 7fb1597fa700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb14c030000 0x7fb14c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.307+0000 7fb1597fa700 1 -- 172.21.15.46:0/3514814619 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb150040f50 con 0x7fb15c0f5140 2024-01-21T08:05:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.307+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15c045ad0 con 0x7fb15c0f5140 2024-01-21T08:05:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.307+0000 7fb15bfff700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb14c030000 0x7fb14c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.307+0000 7fb15bfff700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb14c030000 0x7fb14c0324c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb1480096a0 tx=0x7fb148006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:52.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.313+0000 7fb1597fa700 1 -- 172.21.15.46:0/3514814619 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb150012920 con 0x7fb15c0f5140 2024-01-21T08:05:52.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:52.505+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb15c109f60 con 0x7fb15c0f5140 2024-01-21T08:05:53.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.038+0000 7fb1597fa700 1 -- 172.21.15.46:0/3514814619 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb15002b030 con 0x7fb15c0f5140 2024-01-21T08:05:53.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb14c030000 msgr2=0x7fb14c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:53.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb14c030000 0x7fb14c0324c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb1480096a0 tx=0x7fb148006b40).stop 2024-01-21T08:05:53.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 msgr2=0x7fb15c1086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:53.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c1086c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb150013510 tx=0x7fb15000b040).stop 2024-01-21T08:05:53.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 shutdown_connections 2024-01-21T08:05:53.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb14c030000 0x7fb14c0324c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:53.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 --2- 172.21.15.46:0/3514814619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb15c0f5140 0x7fb15c1086c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:53.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.039+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 >> 172.21.15.46:0/3514814619 conn(0x7fb15c0f08c0 msgr2=0x7fb15c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:53.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.040+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 shutdown_connections 2024-01-21T08:05:53.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:05:53.040+0000 7fb161e4e700 1 -- 172.21.15.46:0/3514814619 wait complete. 2024-01-21T08:05:53.042 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-01-21T08:05:53.339 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:52.506865+0000 mon.smithi046 (mon.0) 127 : audit [INF] from='client.? 172.21.15.46:0/3514814619' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-21T08:05:53.417 INFO:tasks.cephadm:Adding mon.smithi046 on smithi046 2024-01-21T08:05:53.417 INFO:tasks.cephadm:Adding mon.smithi078 on smithi078 2024-01-21T08:05:53.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch apply mon '2;smithi046:172.21.15.46=smithi046;smithi078:172.21.15.78=smithi078' 2024-01-21T08:05:54.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:52.593719+0000 mgr.smithi046.lmxnwc (mgr.14164) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:54.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:53.038400+0000 mon.smithi046 (mon.0) 128 : audit [INF] from='client.? 172.21.15.46:0/3514814619' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-21T08:05:54.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:53.038443+0000 mon.smithi046 (mon.0) 129 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T08:05:54.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 -- 172.21.15.78:0/2368231688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 msgr2=0x7f76740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 --2- 172.21.15.78:0/2368231688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f76740f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f7664009fb0 tx=0x7f7664009b30).stop 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 -- 172.21.15.78:0/2368231688 shutdown_connections 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 --2- 172.21.15.78:0/2368231688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f76740f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 -- 172.21.15.78:0/2368231688 >> 172.21.15.78:0/2368231688 conn(0x7f76740f0b70 msgr2=0x7f76740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 -- 172.21.15.78:0/2368231688 shutdown_connections 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 -- 172.21.15.78:0/2368231688 wait complete. 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.619+0000 7f767b84f700 1 Processor -- start 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f767b84f700 1 -- start start 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f767b84f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f7674105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f767b84f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7674105b20 con 0x7f76740f5410 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f76795eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f7674105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f76795eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f7674105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:46722/0 (socket says 172.21.15.78:46722) 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f76795eb700 1 -- 172.21.15.78:0/2908745454 learned_addr learned my addr 172.21.15.78:0/2908745454 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f76795eb700 1 -- 172.21.15.78:0/2908745454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664005040 con 0x7f76740f5410 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f76795eb700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f7674105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f7664013510 tx=0x7f766400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f7669ffb700 1 -- 172.21.15.78:0/2908745454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7664009320 con 0x7f76740f5410 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f7669ffb700 1 -- 172.21.15.78:0/2908745454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7664013f00 con 0x7f76740f5410 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7674105d20 con 0x7f76740f5410 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.620+0000 7f7669ffb700 1 -- 172.21.15.78:0/2908745454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7664014d90 con 0x7f76740f5410 2024-01-21T08:05:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.621+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76741061b0 con 0x7f76740f5410 2024-01-21T08:05:54.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.621+0000 7f7669ffb700 1 -- 172.21.15.78:0/2908745454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7664014420 con 0x7f76740f5410 2024-01-21T08:05:54.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.621+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7674045ad0 con 0x7f76740f5410 2024-01-21T08:05:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.621+0000 7f7669ffb700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f766002ff60 0x7f7660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.622+0000 7f7669ffb700 1 -- 172.21.15.78:0/2908745454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7664041000 con 0x7f76740f5410 2024-01-21T08:05:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.622+0000 7f7678dea700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f766002ff60 0x7f7660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.622+0000 7f7678dea700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f766002ff60 0x7f7660032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f76700096a0 tx=0x7f7670006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.626+0000 7f7669ffb700 1 -- 172.21.15.78:0/2908745454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f766406d2b0 con 0x7f76740f5410 2024-01-21T08:05:54.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.801+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi046:172.21.15.46=smithi046;smithi078:172.21.15.78=smithi078", "target": ["mon-mgr", ""]}) v1 -- 0x7f76740f1e20 con 0x7f766002ff60 2024-01-21T08:05:54.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.807+0000 7f7669ffb700 1 -- 172.21.15.78:0/2908745454 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f76740f1e20 con 0x7f766002ff60 2024-01-21T08:05:54.809 INFO:teuthology.orchestra.run.smithi078.stdout:Scheduled mon update... 2024-01-21T08:05:54.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f766002ff60 msgr2=0x7f7660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f766002ff60 0x7f7660032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f76700096a0 tx=0x7f7670006b40).stop 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 msgr2=0x7f7674105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f7674105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f7664013510 tx=0x7f766400b040).stop 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 shutdown_connections 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f766002ff60 0x7f7660032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 --2- 172.21.15.78:0/2908745454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76740f5410 0x7f7674105630 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 >> 172.21.15.78:0/2908745454 conn(0x7f76740f0b70 msgr2=0x7f76740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 shutdown_connections 2024-01-21T08:05:54.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:54.809+0000 7f767b84f700 1 -- 172.21.15.78:0/2908745454 wait complete. 2024-01-21T08:05:55.195 DEBUG:teuthology.orchestra.run.smithi078:mon.smithi078> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi078.service 2024-01-21T08:05:55.197 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:05:55.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:05:55.224 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:-- Logs begin at Sun 2024-01-21 07:53:33 UTC. -- 2024-01-21T08:05:56.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:54.593932+0000 mgr.smithi046.lmxnwc (mgr.14164) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:56.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:54.804102+0000 mgr.smithi046.lmxnwc (mgr.14164) 20 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi046:172.21.15.46=smithi046;smithi078:172.21.15.78=smithi078", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:05:56.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:05:54.805405+0000 mgr.smithi046.lmxnwc (mgr.14164) 21 : cephadm [INF] Saving service mon spec with placement smithi046:172.21.15.46=smithi046;smithi078:172.21.15.78=smithi078;count:2 2024-01-21T08:05:56.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:54.807840+0000 mon.smithi046 (mon.0) 130 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:05:56.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.428+0000 7f7878c54700 1 -- 172.21.15.78:0/1528309770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 msgr2=0x7f78740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.428+0000 7f7878c54700 1 --2- 172.21.15.78:0/1528309770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f78740f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f786c009fb0 tx=0x7f786c009b30).stop 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.428+0000 7f7878c54700 1 -- 172.21.15.78:0/1528309770 shutdown_connections 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.428+0000 7f7878c54700 1 --2- 172.21.15.78:0/1528309770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f78740f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.428+0000 7f7878c54700 1 -- 172.21.15.78:0/1528309770 >> 172.21.15.78:0/1528309770 conn(0x7f78740f0b70 msgr2=0x7f78740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.428+0000 7f7878c54700 1 -- 172.21.15.78:0/1528309770 shutdown_connections 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f7878c54700 1 -- 172.21.15.78:0/1528309770 wait complete. 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f7878c54700 1 Processor -- start 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f7878c54700 1 -- start start 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f7878c54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f7874108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f7878c54700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7874108e10 con 0x7f78740f5410 2024-01-21T08:05:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f787259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f7874108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f787259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f7874108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:46730/0 (socket says 172.21.15.78:46730) 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.429+0000 7f787259c700 1 -- 172.21.15.78:0/3524096244 learned_addr learned my addr 172.21.15.78:0/3524096244 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.430+0000 7f787259c700 1 -- 172.21.15.78:0/3524096244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c005040 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.430+0000 7f787259c700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f7874108920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f786c013510 tx=0x7f786c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.430+0000 7f786affd700 1 -- 172.21.15.78:0/3524096244 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f786c009320 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.430+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7874109010 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.430+0000 7f786affd700 1 -- 172.21.15.78:0/3524096244 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f786c013f00 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.430+0000 7f786affd700 1 -- 172.21.15.78:0/3524096244 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f786c014d90 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.430+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78741094a0 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.431+0000 7f786affd700 1 -- 172.21.15.78:0/3524096244 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f786c014420 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.431+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7874045ad0 con 0x7f78740f5410 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.431+0000 7f786affd700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f786002ff60 0x7f7860032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.431+0000 7f786affd700 1 -- 172.21.15.78:0/3524096244 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f786c041070 con 0x7f78740f5410 2024-01-21T08:05:56.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.431+0000 7f7871d9b700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f786002ff60 0x7f7860032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:56.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.431+0000 7f7871d9b700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f786002ff60 0x7f7860032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f785c0096a0 tx=0x7f785c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:56.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.435+0000 7f786affd700 1 -- 172.21.15.78:0/3524096244 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f786c026bf0 con 0x7f78740f5410 2024-01-21T08:05:56.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.642+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f787410a350 con 0x7f78740f5410 2024-01-21T08:05:56.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.643+0000 7f786affd700 1 -- 172.21.15.78:0/3524096244 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f786c01db70 con 0x7f78740f5410 2024-01-21T08:05:56.645 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:05:56.645 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:05:56.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.644+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f786002ff60 msgr2=0x7f7860032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.644+0000 7f7878c54700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f786002ff60 0x7f7860032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f785c0096a0 tx=0x7f785c006b40).stop 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.644+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 msgr2=0x7f7874108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.644+0000 7f7878c54700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f7874108920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f786c013510 tx=0x7f786c00b040).stop 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.645+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 shutdown_connections 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.645+0000 7f7878c54700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f786002ff60 0x7f7860032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.645+0000 7f7878c54700 1 --2- 172.21.15.78:0/3524096244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78740f5410 0x7f7874108920 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.645+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 >> 172.21.15.78:0/3524096244 conn(0x7f78740f0b70 msgr2=0x7f787405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.645+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 shutdown_connections 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:56.645+0000 7f7878c54700 1 -- 172.21.15.78:0/3524096244 wait complete. 2024-01-21T08:05:56.647 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:05:57.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:56.643886+0000 mon.smithi046 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.78:0/3524096244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:05:58.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:05:58.065 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:05:58.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:05:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:56.594133+0000 mgr.smithi046.lmxnwc (mgr.14164) 22 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:05:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.559+0000 7f9eceea7700 1 -- 172.21.15.78:0/300224489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 msgr2=0x7f9ec80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.559+0000 7f9eceea7700 1 --2- 172.21.15.78:0/300224489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec80f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9ebc009fb0 tx=0x7f9ebc009b30).stop 2024-01-21T08:05:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.559+0000 7f9eceea7700 1 -- 172.21.15.78:0/300224489 shutdown_connections 2024-01-21T08:05:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.559+0000 7f9eceea7700 1 --2- 172.21.15.78:0/300224489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec80f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.559+0000 7f9eceea7700 1 -- 172.21.15.78:0/300224489 >> 172.21.15.78:0/300224489 conn(0x7f9ec80f1c70 msgr2=0x7f9ec80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.559+0000 7f9eceea7700 1 -- 172.21.15.78:0/300224489 shutdown_connections 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.559+0000 7f9eceea7700 1 -- 172.21.15.78:0/300224489 wait complete. 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.560+0000 7f9eceea7700 1 Processor -- start 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.560+0000 7f9eceea7700 1 -- start start 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.560+0000 7f9eceea7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.560+0000 7f9eceea7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8106c20 con 0x7f9ec80f6510 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.560+0000 7f9eccc43700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.560+0000 7f9eccc43700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:46752/0 (socket says 172.21.15.78:46752) 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.560+0000 7f9eccc43700 1 -- 172.21.15.78:0/4034347666 learned_addr learned my addr 172.21.15.78:0/4034347666 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.561+0000 7f9eccc43700 1 -- 172.21.15.78:0/4034347666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc005040 con 0x7f9ec80f6510 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.561+0000 7f9eccc43700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec8106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9ebc014000 tx=0x7f9ebc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.561+0000 7f9ec57fa700 1 -- 172.21.15.78:0/4034347666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc0095f0 con 0x7f9ec80f6510 2024-01-21T08:05:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.561+0000 7f9ec57fa700 1 -- 172.21.15.78:0/4034347666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ebc007e70 con 0x7f9ec80f6510 2024-01-21T08:05:59.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.561+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec8106e20 con 0x7f9ec80f6510 2024-01-21T08:05:59.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.561+0000 7f9ec57fa700 1 -- 172.21.15.78:0/4034347666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc014ea0 con 0x7f9ec80f6510 2024-01-21T08:05:59.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.561+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec81072b0 con 0x7f9ec80f6510 2024-01-21T08:05:59.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.562+0000 7f9ec57fa700 1 -- 172.21.15.78:0/4034347666 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ebc01d4d0 con 0x7f9ec80f6510 2024-01-21T08:05:59.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.562+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec8045ad0 con 0x7f9ec80f6510 2024-01-21T08:05:59.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.562+0000 7f9ec57fa700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9eb802ffb0 0x7f9eb8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:05:59.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.562+0000 7f9ec57fa700 1 -- 172.21.15.78:0/4034347666 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ebc040c30 con 0x7f9ec80f6510 2024-01-21T08:05:59.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.562+0000 7f9ec7fff700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9eb802ffb0 0x7f9eb8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:05:59.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.562+0000 7f9ec7fff700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9eb802ffb0 0x7f9eb8032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9eb40096a0 tx=0x7f9eb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:05:59.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.567+0000 7f9ec57fa700 1 -- 172.21.15.78:0/4034347666 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ebc01e410 con 0x7f9ec80f6510 2024-01-21T08:05:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.774+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ec80fbf60 con 0x7f9ec80f6510 2024-01-21T08:05:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.775+0000 7f9ec57fa700 1 -- 172.21.15.78:0/4034347666 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ebc00f300 con 0x7f9ec80f6510 2024-01-21T08:05:59.777 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:05:59.777 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:05:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9eb802ffb0 msgr2=0x7f9eb8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9eb802ffb0 0x7f9eb8032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9eb40096a0 tx=0x7f9eb4006b40).stop 2024-01-21T08:05:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 msgr2=0x7f9ec8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:05:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec8106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9ebc014000 tx=0x7f9ebc00b040).stop 2024-01-21T08:05:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 shutdown_connections 2024-01-21T08:05:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9eb802ffb0 0x7f9eb8032470 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 --2- 172.21.15.78:0/4034347666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ec80f6510 0x7f9ec8106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:05:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 >> 172.21.15.78:0/4034347666 conn(0x7f9ec80f1c70 msgr2=0x7f9ec80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:05:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 shutdown_connections 2024-01-21T08:05:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:05:59.776+0000 7f9eceea7700 1 -- 172.21.15.78:0/4034347666 wait complete. 2024-01-21T08:05:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:00.702 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:05:58.594408+0000 mgr.smithi046.lmxnwc (mgr.14164) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:00.702 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:05:59.775717+0000 mon.smithi046 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.78:0/4034347666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:01.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:01.172 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:02.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.422+0000 7f8ed4477700 1 -- 172.21.15.78:0/3573282228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 msgr2=0x7f8ecc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:02.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.422+0000 7f8ed4477700 1 --2- 172.21.15.78:0/3573282228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc0f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8ebc009fb0 tx=0x7f8ebc009b30).stop 2024-01-21T08:06:02.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.422+0000 7f8ed4477700 1 -- 172.21.15.78:0/3573282228 shutdown_connections 2024-01-21T08:06:02.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.422+0000 7f8ed4477700 1 --2- 172.21.15.78:0/3573282228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc0f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:02.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.422+0000 7f8ed4477700 1 -- 172.21.15.78:0/3573282228 >> 172.21.15.78:0/3573282228 conn(0x7f8ecc0f1c70 msgr2=0x7f8ecc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:02.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.422+0000 7f8ed4477700 1 -- 172.21.15.78:0/3573282228 shutdown_connections 2024-01-21T08:06:02.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed4477700 1 -- 172.21.15.78:0/3573282228 wait complete. 2024-01-21T08:06:02.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed4477700 1 Processor -- start 2024-01-21T08:06:02.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed4477700 1 -- start start 2024-01-21T08:06:02.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed4477700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:02.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed4477700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc106c20 con 0x7f8ecc0f6510 2024-01-21T08:06:02.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed2213700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:02.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed2213700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:41018/0 (socket says 172.21.15.78:41018) 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.423+0000 7f8ed2213700 1 -- 172.21.15.78:0/2393289602 learned_addr learned my addr 172.21.15.78:0/2393289602 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.424+0000 7f8ed2213700 1 -- 172.21.15.78:0/2393289602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc005040 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.424+0000 7f8ed2213700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f8ebc014000 tx=0x7f8ebc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.424+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/2393289602 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ebc0095f0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.424+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/2393289602 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ebc007e70 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.424+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc106e20 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.424+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/2393289602 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ebc014ea0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.424+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc1072b0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.425+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/2393289602 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ebc01d4e0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.425+0000 7f8ec2ffd700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8eb802ff60 0x7f8eb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.425+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/2393289602 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ebc0408f0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.425+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc045ad0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.425+0000 7f8ed1a12700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8eb802ff60 0x7f8eb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:02.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.425+0000 7f8ed1a12700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8eb802ff60 0x7f8eb8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8ec80096a0 tx=0x7f8ec8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:02.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.429+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/2393289602 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ebc00f8f0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:00.594700+0000 mgr.smithi046.lmxnwc (mgr.14164) 24 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:02.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:01.541218+0000 mon.smithi046 (mon.0) 133 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:06:02.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.633+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ecc1080b0 con 0x7f8ecc0f6510 2024-01-21T08:06:02.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.634+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/2393289602 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ebc02c090 con 0x7f8ecc0f6510 2024-01-21T08:06:02.636 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:02.636 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8eb802ff60 msgr2=0x7f8eb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8eb802ff60 0x7f8eb8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8ec80096a0 tx=0x7f8ec8006b40).stop 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 msgr2=0x7f8ecc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f8ebc014000 tx=0x7f8ebc00b040).stop 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 shutdown_connections 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8eb802ff60 0x7f8eb8032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 --2- 172.21.15.78:0/2393289602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ecc0f6510 0x7f8ecc106730 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.635+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 >> 172.21.15.78:0/2393289602 conn(0x7f8ecc0f1c70 msgr2=0x7f8ecc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.636+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 shutdown_connections 2024-01-21T08:06:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:02.636+0000 7f8ed4477700 1 -- 172.21.15.78:0/2393289602 wait complete. 2024-01-21T08:06:02.639 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:06:01.542543+0000 mgr.smithi046.lmxnwc (mgr.14164) 25 : cephadm [INF] Deploying daemon node-exporter.smithi046 on smithi046 2024-01-21T08:06:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:02.635092+0000 mon.smithi046 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.78:0/2393289602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:04.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:04.026 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:04.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:02.594964+0000 mgr.smithi046.lmxnwc (mgr.14164) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.302+0000 7fdf25819700 1 -- 172.21.15.78:0/2807227998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 msgr2=0x7fdf200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.302+0000 7fdf25819700 1 --2- 172.21.15.78:0/2807227998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf200f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdf10009fb0 tx=0x7fdf10009b30).stop 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.302+0000 7fdf25819700 1 -- 172.21.15.78:0/2807227998 shutdown_connections 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.302+0000 7fdf25819700 1 --2- 172.21.15.78:0/2807227998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf200f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.302+0000 7fdf25819700 1 -- 172.21.15.78:0/2807227998 >> 172.21.15.78:0/2807227998 conn(0x7fdf200f1c70 msgr2=0x7fdf200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.302+0000 7fdf25819700 1 -- 172.21.15.78:0/2807227998 shutdown_connections 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.302+0000 7fdf25819700 1 -- 172.21.15.78:0/2807227998 wait complete. 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf25819700 1 Processor -- start 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf25819700 1 -- start start 2024-01-21T08:06:05.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf25819700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf20108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:05.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf25819700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf20108e10 con 0x7fdf200f6510 2024-01-21T08:06:05.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf20108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:05.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf20108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:41036/0 (socket says 172.21.15.78:41036) 2024-01-21T08:06:05.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf1effd700 1 -- 172.21.15.78:0/2609235650 learned_addr learned my addr 172.21.15.78:0/2609235650 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:05.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.303+0000 7fdf1effd700 1 -- 172.21.15.78:0/2609235650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10005040 con 0x7fdf200f6510 2024-01-21T08:06:05.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.304+0000 7fdf1effd700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf20108920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fdf10013510 tx=0x7fdf1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:05.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.304+0000 7fdf077fe700 1 -- 172.21.15.78:0/2609235650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf10009320 con 0x7fdf200f6510 2024-01-21T08:06:05.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.304+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf20109010 con 0x7fdf200f6510 2024-01-21T08:06:05.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.304+0000 7fdf077fe700 1 -- 172.21.15.78:0/2609235650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf10013f00 con 0x7fdf200f6510 2024-01-21T08:06:05.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.304+0000 7fdf077fe700 1 -- 172.21.15.78:0/2609235650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf10014d90 con 0x7fdf200f6510 2024-01-21T08:06:05.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.304+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf201094a0 con 0x7fdf200f6510 2024-01-21T08:06:05.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.305+0000 7fdf077fe700 1 -- 172.21.15.78:0/2609235650 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf10014420 con 0x7fdf200f6510 2024-01-21T08:06:05.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.305+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf20045ad0 con 0x7fdf200f6510 2024-01-21T08:06:05.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.305+0000 7fdf077fe700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf0802ff60 0x7fdf08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:05.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.305+0000 7fdf077fe700 1 -- 172.21.15.78:0/2609235650 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf10040ff0 con 0x7fdf200f6510 2024-01-21T08:06:05.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.305+0000 7fdf1e7fc700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf0802ff60 0x7fdf08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:05.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.305+0000 7fdf1e7fc700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf0802ff60 0x7fdf08032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdf140096a0 tx=0x7fdf14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:05.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.310+0000 7fdf077fe700 1 -- 172.21.15.78:0/2609235650 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf100095f0 con 0x7fdf200f6510 2024-01-21T08:06:05.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.514+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf2010a350 con 0x7fdf200f6510 2024-01-21T08:06:05.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.515+0000 7fdf077fe700 1 -- 172.21.15.78:0/2609235650 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf1001db70 con 0x7fdf200f6510 2024-01-21T08:06:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:05.517 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf0802ff60 msgr2=0x7fdf08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf0802ff60 0x7fdf08032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdf140096a0 tx=0x7fdf14006b40).stop 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 msgr2=0x7fdf20108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf20108920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fdf10013510 tx=0x7fdf1000b040).stop 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 shutdown_connections 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf0802ff60 0x7fdf08032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 --2- 172.21.15.78:0/2609235650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf200f6510 0x7fdf20108920 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 >> 172.21.15.78:0/2609235650 conn(0x7fdf200f1c70 msgr2=0x7fdf200f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:05.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 shutdown_connections 2024-01-21T08:06:05.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:05.517+0000 7fdf25819700 1 -- 172.21.15.78:0/2609235650 wait complete. 2024-01-21T08:06:05.520 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:04.595299+0000 mgr.smithi046.lmxnwc (mgr.14164) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:05.516221+0000 mon.smithi046 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.78:0/2609235650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:06.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:06.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 -- 172.21.15.78:0/1326236894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 msgr2=0x7f4d200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 --2- 172.21.15.78:0/1326236894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d200f90f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f4d08009fb0 tx=0x7f4d08009b30).stop 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 -- 172.21.15.78:0/1326236894 shutdown_connections 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 --2- 172.21.15.78:0/1326236894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d200f90f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 -- 172.21.15.78:0/1326236894 >> 172.21.15.78:0/1326236894 conn(0x7f4d200f0b70 msgr2=0x7f4d200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 -- 172.21.15.78:0/1326236894 shutdown_connections 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 -- 172.21.15.78:0/1326236894 wait complete. 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.144+0000 7f4d26050700 1 Processor -- start 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.145+0000 7f4d26050700 1 -- start start 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.145+0000 7f4d26050700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d201066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.145+0000 7f4d26050700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20106bd0 con 0x7f4d20060280 2024-01-21T08:06:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.145+0000 7f4d1f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d201066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.145+0000 7f4d1f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d201066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:41044/0 (socket says 172.21.15.78:41044) 2024-01-21T08:06:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.145+0000 7f4d1f7fe700 1 -- 172.21.15.78:0/1228734723 learned_addr learned my addr 172.21.15.78:0/1228734723 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.145+0000 7f4d1f7fe700 1 -- 172.21.15.78:0/1228734723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08005040 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.146+0000 7f4d1f7fe700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d201066e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f4d08013510 tx=0x7f4d0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.146+0000 7f4d17fff700 1 -- 172.21.15.78:0/1228734723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d08009320 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.146+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d20106dd0 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.146+0000 7f4d17fff700 1 -- 172.21.15.78:0/1228734723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d08013f00 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.146+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d20107260 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.146+0000 7f4d17fff700 1 -- 172.21.15.78:0/1228734723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d08014dc0 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.147+0000 7f4d17fff700 1 -- 172.21.15.78:0/1228734723 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d080263a0 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.147+0000 7f4d17fff700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4d0c02ffb0 0x7f4d0c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.147+0000 7f4d17fff700 1 -- 172.21.15.78:0/1228734723 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d08040f60 con 0x7f4d20060280 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.147+0000 7f4d1effd700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4d0c02ffb0 0x7f4d0c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.147+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d04004fa0 con 0x7f4d20060280 2024-01-21T08:06:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.148+0000 7f4d1effd700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4d0c02ffb0 0x7f4d0c032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4d100096a0 tx=0x7f4d10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.152+0000 7f4d17fff700 1 -- 172.21.15.78:0/1228734723 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d08012900 con 0x7f4d20060280 2024-01-21T08:06:08.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.358+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d04005670 con 0x7f4d20060280 2024-01-21T08:06:08.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.358+0000 7f4d17fff700 1 -- 172.21.15.78:0/1228734723 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d0802b020 con 0x7f4d20060280 2024-01-21T08:06:08.361 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:08.361 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4d0c02ffb0 msgr2=0x7f4d0c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4d0c02ffb0 0x7f4d0c032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4d100096a0 tx=0x7f4d10006b40).stop 2024-01-21T08:06:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 msgr2=0x7f4d201066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d201066e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f4d08013510 tx=0x7f4d0800b040).stop 2024-01-21T08:06:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 shutdown_connections 2024-01-21T08:06:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4d0c02ffb0 0x7f4d0c032470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 --2- 172.21.15.78:0/1228734723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d20060280 0x7f4d201066e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 >> 172.21.15.78:0/1228734723 conn(0x7f4d200f0b70 msgr2=0x7f4d200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 shutdown_connections 2024-01-21T08:06:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:08.360+0000 7f4d26050700 1 -- 172.21.15.78:0/1228734723 wait complete. 2024-01-21T08:06:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:06.595678+0000 mgr.smithi046.lmxnwc (mgr.14164) 28 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:09.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:08.359518+0000 mon.smithi046 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.78:0/1228734723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:09.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:09.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:08.595974+0000 mgr.smithi046.lmxnwc (mgr.14164) 29 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.129+0000 7f383c042700 1 -- 172.21.15.78:0/873519625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 msgr2=0x7f38340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.129+0000 7f383c042700 1 --2- 172.21.15.78:0/873519625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f38340f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f3824009fb0 tx=0x7f3824009b30).stop 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.129+0000 7f383c042700 1 -- 172.21.15.78:0/873519625 shutdown_connections 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.129+0000 7f383c042700 1 --2- 172.21.15.78:0/873519625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f38340f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.129+0000 7f383c042700 1 -- 172.21.15.78:0/873519625 >> 172.21.15.78:0/873519625 conn(0x7f38340f1c70 msgr2=0x7f38340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.129+0000 7f383c042700 1 -- 172.21.15.78:0/873519625 shutdown_connections 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.129+0000 7f383c042700 1 -- 172.21.15.78:0/873519625 wait complete. 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f383c042700 1 Processor -- start 2024-01-21T08:06:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f383c042700 1 -- start start 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f383c042700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f3834106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f383c042700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3834106c20 con 0x7f38340f6510 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f3839dde700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f3834106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f3839dde700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f3834106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:41052/0 (socket says 172.21.15.78:41052) 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f3839dde700 1 -- 172.21.15.78:0/1962334040 learned_addr learned my addr 172.21.15.78:0/1962334040 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.130+0000 7f3839dde700 1 -- 172.21.15.78:0/1962334040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824005040 con 0x7f38340f6510 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.131+0000 7f3839dde700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f3834106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3824013510 tx=0x7f382400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.131+0000 7f382a7fc700 1 -- 172.21.15.78:0/1962334040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3824009320 con 0x7f38340f6510 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.131+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3834106e20 con 0x7f38340f6510 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.131+0000 7f382a7fc700 1 -- 172.21.15.78:0/1962334040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3824013f00 con 0x7f38340f6510 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.131+0000 7f382a7fc700 1 -- 172.21.15.78:0/1962334040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3824014d90 con 0x7f38340f6510 2024-01-21T08:06:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.131+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38341072b0 con 0x7f38340f6510 2024-01-21T08:06:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.132+0000 7f382a7fc700 1 -- 172.21.15.78:0/1962334040 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38240095f0 con 0x7f38340f6510 2024-01-21T08:06:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.132+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3834045ad0 con 0x7f38340f6510 2024-01-21T08:06:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.132+0000 7f382a7fc700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f382002ff60 0x7f3820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.132+0000 7f382a7fc700 1 -- 172.21.15.78:0/1962334040 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38240410c0 con 0x7f38340f6510 2024-01-21T08:06:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.132+0000 7f38395dd700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f382002ff60 0x7f3820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.132+0000 7f38395dd700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f382002ff60 0x7f3820032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f38300096a0 tx=0x7f3830006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.137+0000 7f382a7fc700 1 -- 172.21.15.78:0/1962334040 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3824014420 con 0x7f38340f6510 2024-01-21T08:06:12.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.343+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38340fbf60 con 0x7f38340f6510 2024-01-21T08:06:12.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.344+0000 7f382a7fc700 1 -- 172.21.15.78:0/1962334040 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f382401d7f0 con 0x7f38340f6510 2024-01-21T08:06:12.346 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:12.346 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:12.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f382002ff60 msgr2=0x7f3820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:12.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f382002ff60 0x7f3820032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f38300096a0 tx=0x7f3830006b40).stop 2024-01-21T08:06:12.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 msgr2=0x7f3834106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:12.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f3834106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3824013510 tx=0x7f382400b040).stop 2024-01-21T08:06:12.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 shutdown_connections 2024-01-21T08:06:12.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f382002ff60 0x7f3820032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:12.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 --2- 172.21.15.78:0/1962334040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38340f6510 0x7f3834106730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:12.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 >> 172.21.15.78:0/1962334040 conn(0x7f38340f1c70 msgr2=0x7f38340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:12.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 shutdown_connections 2024-01-21T08:06:12.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:12.345+0000 7f383c042700 1 -- 172.21.15.78:0/1962334040 wait complete. 2024-01-21T08:06:12.348 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:10.596260+0000 mgr.smithi046.lmxnwc (mgr.14164) 30 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:12.344805+0000 mon.smithi046 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.78:0/1962334040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:13.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:13.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:14.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:12.596597+0000 mgr.smithi046.lmxnwc (mgr.14164) 31 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 -- 172.21.15.78:0/1069327759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 msgr2=0x7f95640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 --2- 172.21.15.78:0/1069327759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95640f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9558009fb0 tx=0x7f9558009b30).stop 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 -- 172.21.15.78:0/1069327759 shutdown_connections 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 --2- 172.21.15.78:0/1069327759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95640f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 -- 172.21.15.78:0/1069327759 >> 172.21.15.78:0/1069327759 conn(0x7f95640f0b70 msgr2=0x7f95640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 -- 172.21.15.78:0/1069327759 shutdown_connections 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 -- 172.21.15.78:0/1069327759 wait complete. 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.961+0000 7f956a82e700 1 Processor -- start 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f956a82e700 1 -- start start 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f956a82e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95641088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f956a82e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564108dc0 con 0x7f95640f5410 2024-01-21T08:06:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f9563fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95641088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f9563fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95641088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:43646/0 (socket says 172.21.15.78:43646) 2024-01-21T08:06:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f9563fff700 1 -- 172.21.15.78:0/3397427766 learned_addr learned my addr 172.21.15.78:0/3397427766 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f9563fff700 1 -- 172.21.15.78:0/3397427766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9558005040 con 0x7f95640f5410 2024-01-21T08:06:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f9563fff700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95641088d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9558013510 tx=0x7f955800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.962+0000 7f9560ff9700 1 -- 172.21.15.78:0/3397427766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9558009320 con 0x7f95640f5410 2024-01-21T08:06:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.963+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9564108fc0 con 0x7f95640f5410 2024-01-21T08:06:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.963+0000 7f9560ff9700 1 -- 172.21.15.78:0/3397427766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9558013f00 con 0x7f95640f5410 2024-01-21T08:06:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.963+0000 7f9560ff9700 1 -- 172.21.15.78:0/3397427766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9558014d90 con 0x7f95640f5410 2024-01-21T08:06:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.963+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9564109450 con 0x7f95640f5410 2024-01-21T08:06:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.963+0000 7f9560ff9700 1 -- 172.21.15.78:0/3397427766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95580263a0 con 0x7f95640f5410 2024-01-21T08:06:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.963+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9564045ad0 con 0x7f95640f5410 2024-01-21T08:06:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.963+0000 7f9560ff9700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f954c02ffb0 0x7f954c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.964+0000 7f9560ff9700 1 -- 172.21.15.78:0/3397427766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95580413b0 con 0x7f95640f5410 2024-01-21T08:06:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.964+0000 7f95637fe700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f954c02ffb0 0x7f954c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.964+0000 7f95637fe700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f954c02ffb0 0x7f954c032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f95540096a0 tx=0x7f9554006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:14.968+0000 7f9560ff9700 1 -- 172.21.15.78:0/3397427766 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f955801d380 con 0x7f95640f5410 2024-01-21T08:06:15.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.176+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95640fae60 con 0x7f95640f5410 2024-01-21T08:06:15.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.176+0000 7f9560ff9700 1 -- 172.21.15.78:0/3397427766 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f955801c970 con 0x7f95640f5410 2024-01-21T08:06:15.179 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f954c02ffb0 msgr2=0x7f954c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f954c02ffb0 0x7f954c032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f95540096a0 tx=0x7f9554006b40).stop 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 msgr2=0x7f95641088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95641088d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9558013510 tx=0x7f955800b040).stop 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 shutdown_connections 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f954c02ffb0 0x7f954c032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 --2- 172.21.15.78:0/3397427766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95640f5410 0x7f95641088d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:15.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 >> 172.21.15.78:0/3397427766 conn(0x7f95640f0b70 msgr2=0x7f956405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:15.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 shutdown_connections 2024-01-21T08:06:15.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:15.178+0000 7f956a82e700 1 -- 172.21.15.78:0/3397427766 wait complete. 2024-01-21T08:06:15.181 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:15.177576+0000 mon.smithi046 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.78:0/3397427766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:16.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:16.583 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:14.596906+0000 mgr.smithi046.lmxnwc (mgr.14164) 32 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:18.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:16.597257+0000 mgr.smithi046.lmxnwc (mgr.14164) 33 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:18.597563+0000 mgr.smithi046.lmxnwc (mgr.14164) 34 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:20.739169+0000 mon.smithi046 (mon.0) 139 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:06:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:20.742972+0000 mon.smithi046 (mon.0) 140 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:06:21.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 -- 172.21.15.78:0/1904683848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 msgr2=0x7fbe700f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:21.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 --2- 172.21.15.78:0/1904683848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe700f66e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fbe64009fb0 tx=0x7fbe64009b30).stop 2024-01-21T08:06:21.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 -- 172.21.15.78:0/1904683848 shutdown_connections 2024-01-21T08:06:21.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 --2- 172.21.15.78:0/1904683848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe700f66e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:21.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 -- 172.21.15.78:0/1904683848 >> 172.21.15.78:0/1904683848 conn(0x7fbe700f1ac0 msgr2=0x7fbe700f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:21.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 -- 172.21.15.78:0/1904683848 shutdown_connections 2024-01-21T08:06:21.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 -- 172.21.15.78:0/1904683848 wait complete. 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.734+0000 7fbe749fa700 1 Processor -- start 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.735+0000 7fbe749fa700 1 -- start start 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.735+0000 7fbe749fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe70106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.735+0000 7fbe749fa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70106a10 con 0x7fbe700f62c0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.735+0000 7fbe6f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe70106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.735+0000 7fbe6f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe70106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:43658/0 (socket says 172.21.15.78:43658) 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.735+0000 7fbe6f7fe700 1 -- 172.21.15.78:0/1867643097 learned_addr learned my addr 172.21.15.78:0/1867643097 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.735+0000 7fbe6f7fe700 1 -- 172.21.15.78:0/1867643097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64005040 con 0x7fbe700f62c0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.736+0000 7fbe6f7fe700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe70106520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fbe64013510 tx=0x7fbe6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.736+0000 7fbe57fff700 1 -- 172.21.15.78:0/1867643097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe64009320 con 0x7fbe700f62c0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.736+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70106c10 con 0x7fbe700f62c0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.736+0000 7fbe57fff700 1 -- 172.21.15.78:0/1867643097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe64013f00 con 0x7fbe700f62c0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.736+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe701070a0 con 0x7fbe700f62c0 2024-01-21T08:06:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.736+0000 7fbe57fff700 1 -- 172.21.15.78:0/1867643097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe64014dc0 con 0x7fbe700f62c0 2024-01-21T08:06:21.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.737+0000 7fbe57fff700 1 -- 172.21.15.78:0/1867643097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe640263a0 con 0x7fbe700f62c0 2024-01-21T08:06:21.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.737+0000 7fbe57fff700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe5802ffb0 0x7fbe58032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:21.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.737+0000 7fbe57fff700 1 -- 172.21.15.78:0/1867643097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe64040fe0 con 0x7fbe700f62c0 2024-01-21T08:06:21.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.737+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe70045ad0 con 0x7fbe700f62c0 2024-01-21T08:06:21.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.737+0000 7fbe6effd700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe5802ffb0 0x7fbe58032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:21.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.737+0000 7fbe6effd700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe5802ffb0 0x7fbe58032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fbe600096a0 tx=0x7fbe60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:21.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.742+0000 7fbe57fff700 1 -- 172.21.15.78:0/1867643097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe64012920 con 0x7fbe700f62c0 2024-01-21T08:06:21.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.948+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe70107dc0 con 0x7fbe700f62c0 2024-01-21T08:06:21.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.948+0000 7fbe57fff700 1 -- 172.21.15.78:0/1867643097 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe6402b020 con 0x7fbe700f62c0 2024-01-21T08:06:21.951 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:21.951 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:21.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe5802ffb0 msgr2=0x7fbe58032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:21.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe5802ffb0 0x7fbe58032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fbe600096a0 tx=0x7fbe60006b40).stop 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 msgr2=0x7fbe70106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe70106520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fbe64013510 tx=0x7fbe6400b040).stop 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 shutdown_connections 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe5802ffb0 0x7fbe58032470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 --2- 172.21.15.78:0/1867643097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe700f62c0 0x7fbe70106520 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 >> 172.21.15.78:0/1867643097 conn(0x7fbe700f1ac0 msgr2=0x7fbe700f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 shutdown_connections 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:21.950+0000 7fbe749fa700 1 -- 172.21.15.78:0/1867643097 wait complete. 2024-01-21T08:06:21.953 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:20.597938+0000 mgr.smithi046.lmxnwc (mgr.14164) 35 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:21.949412+0000 mon.smithi046 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.78:0/1867643097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:22.598363+0000 mgr.smithi046.lmxnwc (mgr.14164) 36 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:24.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:24.789 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.979+0000 7f4e06d8a700 1 -- 172.21.15.78:0/4032742700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 msgr2=0x7f4e000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.979+0000 7f4e06d8a700 1 --2- 172.21.15.78:0/4032742700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e000f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f4df4009fb0 tx=0x7f4df4009b30).stop 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.979+0000 7f4e06d8a700 1 -- 172.21.15.78:0/4032742700 shutdown_connections 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.979+0000 7f4e06d8a700 1 --2- 172.21.15.78:0/4032742700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e000f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.979+0000 7f4e06d8a700 1 -- 172.21.15.78:0/4032742700 >> 172.21.15.78:0/4032742700 conn(0x7f4e000f0b70 msgr2=0x7f4e000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.979+0000 7f4e06d8a700 1 -- 172.21.15.78:0/4032742700 shutdown_connections 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.979+0000 7f4e06d8a700 1 -- 172.21.15.78:0/4032742700 wait complete. 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e06d8a700 1 Processor -- start 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e06d8a700 1 -- start start 2024-01-21T08:06:25.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e06d8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e00105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e06d8a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e00105b20 con 0x7f4e000f5410 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e04b26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e00105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e04b26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e00105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49202/0 (socket says 172.21.15.78:49202) 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e04b26700 1 -- 172.21.15.78:0/1924901823 learned_addr learned my addr 172.21.15.78:0/1924901823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.980+0000 7f4e04b26700 1 -- 172.21.15.78:0/1924901823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4005040 con 0x7f4e000f5410 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.981+0000 7f4e04b26700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e00105630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f4df4013510 tx=0x7f4df400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.981+0000 7f4dfd7fa700 1 -- 172.21.15.78:0/1924901823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df4009320 con 0x7f4e000f5410 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.981+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e00105d20 con 0x7f4e000f5410 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.981+0000 7f4dfd7fa700 1 -- 172.21.15.78:0/1924901823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4df4013f00 con 0x7f4e000f5410 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.981+0000 7f4dfd7fa700 1 -- 172.21.15.78:0/1924901823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df4014d90 con 0x7f4e000f5410 2024-01-21T08:06:25.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.981+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e001061b0 con 0x7f4e000f5410 2024-01-21T08:06:25.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.982+0000 7f4dfd7fa700 1 -- 172.21.15.78:0/1924901823 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4df40095f0 con 0x7f4e000f5410 2024-01-21T08:06:25.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.982+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e00045ad0 con 0x7f4e000f5410 2024-01-21T08:06:25.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.982+0000 7f4dfd7fa700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4df002ff60 0x7f4df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:25.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.982+0000 7f4dfd7fa700 1 -- 172.21.15.78:0/1924901823 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4df4042d90 con 0x7f4e000f5410 2024-01-21T08:06:25.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.982+0000 7f4dfffff700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4df002ff60 0x7f4df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:25.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.982+0000 7f4dfffff700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4df002ff60 0x7f4df0032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4dec0096a0 tx=0x7f4dec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:25.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:25.987+0000 7f4dfd7fa700 1 -- 172.21.15.78:0/1924901823 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4df401f7f0 con 0x7f4e000f5410 2024-01-21T08:06:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.193+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e000fae60 con 0x7f4e000f5410 2024-01-21T08:06:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.193+0000 7f4dfd7fa700 1 -- 172.21.15.78:0/1924901823 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4df4011290 con 0x7f4e000f5410 2024-01-21T08:06:26.195 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:26.196 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4df002ff60 msgr2=0x7f4df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4df002ff60 0x7f4df0032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4dec0096a0 tx=0x7f4dec006b40).stop 2024-01-21T08:06:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 msgr2=0x7f4e00105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e00105630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f4df4013510 tx=0x7f4df400b040).stop 2024-01-21T08:06:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 shutdown_connections 2024-01-21T08:06:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4df002ff60 0x7f4df0032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 --2- 172.21.15.78:0/1924901823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e000f5410 0x7f4e00105630 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 >> 172.21.15.78:0/1924901823 conn(0x7f4e000f0b70 msgr2=0x7f4e000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 shutdown_connections 2024-01-21T08:06:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:26.195+0000 7f4e06d8a700 1 -- 172.21.15.78:0/1924901823 wait complete. 2024-01-21T08:06:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:24.598769+0000 mgr.smithi046.lmxnwc (mgr.14164) 37 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:26.194418+0000 mon.smithi046 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.78:0/1924901823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:27.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:27.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:26.599163+0000 mgr.smithi046.lmxnwc (mgr.14164) 38 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:28.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.808+0000 7f7efdc9f700 1 -- 172.21.15.78:0/2056187998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 msgr2=0x7f7ef80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:28.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.808+0000 7f7efdc9f700 1 --2- 172.21.15.78:0/2056187998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f9150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f7ee8009fb0 tx=0x7f7ee8009b30).stop 2024-01-21T08:06:28.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.808+0000 7f7efdc9f700 1 -- 172.21.15.78:0/2056187998 shutdown_connections 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.808+0000 7f7efdc9f700 1 --2- 172.21.15.78:0/2056187998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f9150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.808+0000 7f7efdc9f700 1 -- 172.21.15.78:0/2056187998 >> 172.21.15.78:0/2056187998 conn(0x7f7ef80f0b70 msgr2=0x7f7ef80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.808+0000 7f7efdc9f700 1 -- 172.21.15.78:0/2056187998 shutdown_connections 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.808+0000 7f7efdc9f700 1 -- 172.21.15.78:0/2056187998 wait complete. 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.809+0000 7f7efdc9f700 1 Processor -- start 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.809+0000 7f7efdc9f700 1 -- start start 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.809+0000 7f7efdc9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.809+0000 7f7efdc9f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef80f5050 con 0x7f7ef8060220 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7ef77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7ef77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49220/0 (socket says 172.21.15.78:49220) 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7ef77fe700 1 -- 172.21.15.78:0/978252617 learned_addr learned my addr 172.21.15.78:0/978252617 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7ef77fe700 1 -- 172.21.15.78:0/978252617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee8005040 con 0x7f7ef8060220 2024-01-21T08:06:28.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7ef77fe700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f6bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7ee8013510 tx=0x7f7ee800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:28.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7edffff700 1 -- 172.21.15.78:0/978252617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ee8009320 con 0x7f7ef8060220 2024-01-21T08:06:28.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef80f51f0 con 0x7f7ef8060220 2024-01-21T08:06:28.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7edffff700 1 -- 172.21.15.78:0/978252617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ee8013f00 con 0x7f7ef8060220 2024-01-21T08:06:28.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7edffff700 1 -- 172.21.15.78:0/978252617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ee8014dc0 con 0x7f7ef8060220 2024-01-21T08:06:28.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.810+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef80f5680 con 0x7f7ef8060220 2024-01-21T08:06:28.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.811+0000 7f7edffff700 1 -- 172.21.15.78:0/978252617 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ee80095f0 con 0x7f7ef8060220 2024-01-21T08:06:28.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.811+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef8059100 con 0x7f7ef8060220 2024-01-21T08:06:28.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.811+0000 7f7edffff700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7ee002ff60 0x7f7ee0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:28.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.811+0000 7f7edffff700 1 -- 172.21.15.78:0/978252617 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ee80410f0 con 0x7f7ef8060220 2024-01-21T08:06:28.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.811+0000 7f7ef6ffd700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7ee002ff60 0x7f7ee0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:28.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.812+0000 7f7ef6ffd700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7ee002ff60 0x7f7ee0032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7eec0096a0 tx=0x7f7eec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:28.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:28.816+0000 7f7edffff700 1 -- 172.21.15.78:0/978252617 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ee8014420 con 0x7f7ef8060220 2024-01-21T08:06:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.022+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ef80fbf50 con 0x7f7ef8060220 2024-01-21T08:06:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.023+0000 7f7edffff700 1 -- 172.21.15.78:0/978252617 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ee8010230 con 0x7f7ef8060220 2024-01-21T08:06:29.025 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:29.025 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7ee002ff60 msgr2=0x7f7ee0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7ee002ff60 0x7f7ee0032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7eec0096a0 tx=0x7f7eec006b40).stop 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 msgr2=0x7f7ef80f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f6bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7ee8013510 tx=0x7f7ee800b040).stop 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 shutdown_connections 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7ee002ff60 0x7f7ee0032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 --2- 172.21.15.78:0/978252617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef8060220 0x7f7ef80f6bd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 >> 172.21.15.78:0/978252617 conn(0x7f7ef80f0b70 msgr2=0x7f7ef80f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 shutdown_connections 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:29.024+0000 7f7efdc9f700 1 -- 172.21.15.78:0/978252617 wait complete. 2024-01-21T08:06:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:29.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:29.023770+0000 mon.smithi046 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.78:0/978252617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:30.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:30.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:28.599416+0000 mgr.smithi046.lmxnwc (mgr.14164) 39 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.741+0000 7fdd7a41b700 1 -- 172.21.15.78:0/3317449418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 msgr2=0x7fdd740f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.741+0000 7fdd7a41b700 1 --2- 172.21.15.78:0/3317449418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd740f4b60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fdd68009fb0 tx=0x7fdd68009b30).stop 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 -- 172.21.15.78:0/3317449418 shutdown_connections 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 --2- 172.21.15.78:0/3317449418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd740f4b60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 -- 172.21.15.78:0/3317449418 >> 172.21.15.78:0/3317449418 conn(0x7fdd740f0260 msgr2=0x7fdd740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 -- 172.21.15.78:0/3317449418 shutdown_connections 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 -- 172.21.15.78:0/3317449418 wait complete. 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 Processor -- start 2024-01-21T08:06:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 -- start start 2024-01-21T08:06:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd741066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.742+0000 7fdd7a41b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74106bc0 con 0x7fdd740f4740 2024-01-21T08:06:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd73fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd741066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd73fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd741066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49240/0 (socket says 172.21.15.78:49240) 2024-01-21T08:06:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd73fff700 1 -- 172.21.15.78:0/2810568373 learned_addr learned my addr 172.21.15.78:0/2810568373 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd73fff700 1 -- 172.21.15.78:0/2810568373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd68005040 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd73fff700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd741066d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fdd68013510 tx=0x7fdd6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd70ff9700 1 -- 172.21.15.78:0/2810568373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd68009320 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd74106dc0 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd70ff9700 1 -- 172.21.15.78:0/2810568373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd68013f00 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd70ff9700 1 -- 172.21.15.78:0/2810568373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd68014d90 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.743+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd74107250 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.744+0000 7fdd70ff9700 1 -- 172.21.15.78:0/2810568373 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd680095f0 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.744+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd74045ad0 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.744+0000 7fdd70ff9700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdd5c02ff60 0x7fdd5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.744+0000 7fdd70ff9700 1 -- 172.21.15.78:0/2810568373 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd68040fc0 con 0x7fdd740f4740 2024-01-21T08:06:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.745+0000 7fdd737fe700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdd5c02ff60 0x7fdd5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.745+0000 7fdd737fe700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdd5c02ff60 0x7fdd5c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fdd640096a0 tx=0x7fdd64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:31.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.749+0000 7fdd70ff9700 1 -- 172.21.15.78:0/2810568373 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd68014420 con 0x7fdd740f4740 2024-01-21T08:06:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.953+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd741080a0 con 0x7fdd740f4740 2024-01-21T08:06:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.953+0000 7fdd70ff9700 1 -- 172.21.15.78:0/2810568373 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd68010230 con 0x7fdd740f4740 2024-01-21T08:06:31.955 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:31.955 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:31.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdd5c02ff60 msgr2=0x7fdd5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:31.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdd5c02ff60 0x7fdd5c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fdd640096a0 tx=0x7fdd64006b40).stop 2024-01-21T08:06:31.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 msgr2=0x7fdd741066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:31.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd741066d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fdd68013510 tx=0x7fdd6800b040).stop 2024-01-21T08:06:31.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 shutdown_connections 2024-01-21T08:06:31.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdd5c02ff60 0x7fdd5c032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:31.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 --2- 172.21.15.78:0/2810568373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740f4740 0x7fdd741066d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:31.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 >> 172.21.15.78:0/2810568373 conn(0x7fdd740f0260 msgr2=0x7fdd740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:31.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 shutdown_connections 2024-01-21T08:06:31.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:31.955+0000 7fdd7a41b700 1 -- 172.21.15.78:0/2810568373 wait complete. 2024-01-21T08:06:31.958 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:30.599689+0000 mgr.smithi046.lmxnwc (mgr.14164) 40 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:31.954405+0000 mon.smithi046 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.78:0/2810568373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:33.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:33.347 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:34.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.563+0000 7f114c65b700 1 -- 172.21.15.78:0/3146791690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 msgr2=0x7f11440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:34.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.563+0000 7f114c65b700 1 --2- 172.21.15.78:0/3146791690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f11440f6930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f113c009fb0 tx=0x7f113c009b30).stop 2024-01-21T08:06:34.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 -- 172.21.15.78:0/3146791690 shutdown_connections 2024-01-21T08:06:34.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 --2- 172.21.15.78:0/3146791690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f11440f6930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:34.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 -- 172.21.15.78:0/3146791690 >> 172.21.15.78:0/3146791690 conn(0x7f11440f1c70 msgr2=0x7f11440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:34.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 -- 172.21.15.78:0/3146791690 shutdown_connections 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 -- 172.21.15.78:0/3146791690 wait complete. 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 Processor -- start 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 -- start start 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f1144106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114c65b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1144106c10 con 0x7f11440f6510 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114a3f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f1144106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114a3f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f1144106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:47720/0 (socket says 172.21.15.78:47720) 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.564+0000 7f114a3f7700 1 -- 172.21.15.78:0/3154703499 learned_addr learned my addr 172.21.15.78:0/3154703499 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.565+0000 7f114a3f7700 1 -- 172.21.15.78:0/3154703499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c005040 con 0x7f11440f6510 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.565+0000 7f114a3f7700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f1144106720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f113c013510 tx=0x7f113c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.565+0000 7f113affd700 1 -- 172.21.15.78:0/3154703499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f113c009320 con 0x7f11440f6510 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.565+0000 7f113affd700 1 -- 172.21.15.78:0/3154703499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f113c013f00 con 0x7f11440f6510 2024-01-21T08:06:34.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.565+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1144106e10 con 0x7f11440f6510 2024-01-21T08:06:34.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.565+0000 7f113affd700 1 -- 172.21.15.78:0/3154703499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f113c014d90 con 0x7f11440f6510 2024-01-21T08:06:34.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.565+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11441072a0 con 0x7f11440f6510 2024-01-21T08:06:34.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.566+0000 7f113affd700 1 -- 172.21.15.78:0/3154703499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f113c0095f0 con 0x7f11440f6510 2024-01-21T08:06:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.566+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1144045ad0 con 0x7f11440f6510 2024-01-21T08:06:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.566+0000 7f113affd700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f113002ff60 0x7f1130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.566+0000 7f113affd700 1 -- 172.21.15.78:0/3154703499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f113c0410c0 con 0x7f11440f6510 2024-01-21T08:06:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.566+0000 7f1149bf6700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f113002ff60 0x7f1130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.567+0000 7f1149bf6700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f113002ff60 0x7f1130032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f11340096a0 tx=0x7f1134006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:34.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.571+0000 7f113affd700 1 -- 172.21.15.78:0/3154703499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f113c014420 con 0x7f11440f6510 2024-01-21T08:06:34.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:32.599905+0000 mgr.smithi046.lmxnwc (mgr.14164) 41 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.777+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1144108150 con 0x7f11440f6510 2024-01-21T08:06:34.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.777+0000 7f113affd700 1 -- 172.21.15.78:0/3154703499 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f113c01d7f0 con 0x7f11440f6510 2024-01-21T08:06:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:34.780 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:34.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f113002ff60 msgr2=0x7f1130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:34.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f113002ff60 0x7f1130032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f11340096a0 tx=0x7f1134006b40).stop 2024-01-21T08:06:34.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 msgr2=0x7f1144106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:34.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f1144106720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f113c013510 tx=0x7f113c00b040).stop 2024-01-21T08:06:34.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 shutdown_connections 2024-01-21T08:06:34.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f113002ff60 0x7f1130032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:34.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 --2- 172.21.15.78:0/3154703499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11440f6510 0x7f1144106720 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:34.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 >> 172.21.15.78:0/3154703499 conn(0x7f11440f1c70 msgr2=0x7f11440f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:34.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 shutdown_connections 2024-01-21T08:06:34.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:34.779+0000 7f114c65b700 1 -- 172.21.15.78:0/3154703499 wait complete. 2024-01-21T08:06:34.783 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:34.778453+0000 mon.smithi046 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.78:0/3154703499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:36.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:36.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:36.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:34.600247+0000 mgr.smithi046.lmxnwc (mgr.14164) 42 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.420+0000 7f5c69578700 1 -- 172.21.15.78:0/2661560861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 msgr2=0x7f5c640f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.420+0000 7f5c69578700 1 --2- 172.21.15.78:0/2661560861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c640f66c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5c54009fb0 tx=0x7f5c54009b30).stop 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 -- 172.21.15.78:0/2661560861 shutdown_connections 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 --2- 172.21.15.78:0/2661560861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c640f66c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 -- 172.21.15.78:0/2661560861 >> 172.21.15.78:0/2661560861 conn(0x7f5c640f1aa0 msgr2=0x7f5c640f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 -- 172.21.15.78:0/2661560861 shutdown_connections 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 -- 172.21.15.78:0/2661560861 wait complete. 2024-01-21T08:06:37.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 Processor -- start 2024-01-21T08:06:37.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 -- start start 2024-01-21T08:06:37.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:37.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.421+0000 7f5c69578700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c64069e20 con 0x7f5c640f62a0 2024-01-21T08:06:37.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.422+0000 7f5c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.422+0000 7f5c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c6406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:47730/0 (socket says 172.21.15.78:47730) 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.422+0000 7f5c63fff700 1 -- 172.21.15.78:0/2972038157 learned_addr learned my addr 172.21.15.78:0/2972038157 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.422+0000 7f5c63fff700 1 -- 172.21.15.78:0/2972038157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c54005040 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.422+0000 7f5c63fff700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c6406b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5c54013510 tx=0x7f5c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.422+0000 7f5c60ff9700 1 -- 172.21.15.78:0/2972038157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c54009320 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.423+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c64069fc0 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.423+0000 7f5c60ff9700 1 -- 172.21.15.78:0/2972038157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c54013f00 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.423+0000 7f5c60ff9700 1 -- 172.21.15.78:0/2972038157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c54014dc0 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.423+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6406a450 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.423+0000 7f5c60ff9700 1 -- 172.21.15.78:0/2972038157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c540095f0 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.423+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c64059100 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.423+0000 7f5c60ff9700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5c4c02ff60 0x7f5c4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.424+0000 7f5c60ff9700 1 -- 172.21.15.78:0/2972038157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c540410c0 con 0x7f5c640f62a0 2024-01-21T08:06:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.424+0000 7f5c637fe700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5c4c02ff60 0x7f5c4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:37.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.424+0000 7f5c637fe700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5c4c02ff60 0x7f5c4c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5c580096a0 tx=0x7f5c58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:37.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.428+0000 7f5c60ff9700 1 -- 172.21.15.78:0/2972038157 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c5401c790 con 0x7f5c640f62a0 2024-01-21T08:06:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.633+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c640fbce0 con 0x7f5c640f62a0 2024-01-21T08:06:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.633+0000 7f5c60ff9700 1 -- 172.21.15.78:0/2972038157 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c54010230 con 0x7f5c640f62a0 2024-01-21T08:06:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:37.636 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:37.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5c4c02ff60 msgr2=0x7f5c4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:37.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5c4c02ff60 0x7f5c4c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5c580096a0 tx=0x7f5c58006b40).stop 2024-01-21T08:06:37.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 msgr2=0x7f5c6406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:37.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c6406b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5c54013510 tx=0x7f5c5400b040).stop 2024-01-21T08:06:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 shutdown_connections 2024-01-21T08:06:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5c4c02ff60 0x7f5c4c032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 --2- 172.21.15.78:0/2972038157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c640f62a0 0x7f5c6406b9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 >> 172.21.15.78:0/2972038157 conn(0x7f5c640f1aa0 msgr2=0x7f5c640f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 shutdown_connections 2024-01-21T08:06:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:37.635+0000 7f5c69578700 1 -- 172.21.15.78:0/2972038157 wait complete. 2024-01-21T08:06:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:36.600658+0000 mgr.smithi046.lmxnwc (mgr.14164) 43 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:37.634568+0000 mon.smithi046 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.78:0/2972038157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:39.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:39.038 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.241+0000 7f81e37e7700 1 -- 172.21.15.78:0/4027310694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 msgr2=0x7f81dc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.241+0000 7f81e37e7700 1 --2- 172.21.15.78:0/4027310694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc0f90f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f81d8009fb0 tx=0x7f81d8009b30).stop 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.241+0000 7f81e37e7700 1 -- 172.21.15.78:0/4027310694 shutdown_connections 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.241+0000 7f81e37e7700 1 --2- 172.21.15.78:0/4027310694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc0f90f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.241+0000 7f81e37e7700 1 -- 172.21.15.78:0/4027310694 >> 172.21.15.78:0/4027310694 conn(0x7f81dc0f0b70 msgr2=0x7f81dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.241+0000 7f81e37e7700 1 -- 172.21.15.78:0/4027310694 shutdown_connections 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.241+0000 7f81e37e7700 1 -- 172.21.15.78:0/4027310694 wait complete. 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e37e7700 1 Processor -- start 2024-01-21T08:06:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e37e7700 1 -- start start 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e37e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e37e7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc106bd0 con 0x7f81dc060280 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e1583700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e1583700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:47750/0 (socket says 172.21.15.78:47750) 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e1583700 1 -- 172.21.15.78:0/2593546779 learned_addr learned my addr 172.21.15.78:0/2593546779 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.242+0000 7f81e1583700 1 -- 172.21.15.78:0/2593546779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d8005040 con 0x7f81dc060280 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.243+0000 7f81e1583700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc1066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f81d8013510 tx=0x7f81d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.243+0000 7f81d1ffb700 1 -- 172.21.15.78:0/2593546779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d8009320 con 0x7f81dc060280 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.243+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc106dd0 con 0x7f81dc060280 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.243+0000 7f81d1ffb700 1 -- 172.21.15.78:0/2593546779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81d8013f00 con 0x7f81dc060280 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.243+0000 7f81d1ffb700 1 -- 172.21.15.78:0/2593546779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d8014dc0 con 0x7f81dc060280 2024-01-21T08:06:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.243+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc107260 con 0x7f81dc060280 2024-01-21T08:06:40.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.244+0000 7f81d1ffb700 1 -- 172.21.15.78:0/2593546779 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81d8014420 con 0x7f81dc060280 2024-01-21T08:06:40.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.244+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc045ad0 con 0x7f81dc060280 2024-01-21T08:06:40.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.244+0000 7f81d1ffb700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f81c802ff60 0x7f81c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:40.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.244+0000 7f81d1ffb700 1 -- 172.21.15.78:0/2593546779 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81d80410f0 con 0x7f81dc060280 2024-01-21T08:06:40.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.244+0000 7f81e0d82700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f81c802ff60 0x7f81c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:40.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.245+0000 7f81e0d82700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f81c802ff60 0x7f81c8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f81cc0096a0 tx=0x7f81cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:40.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.249+0000 7f81d1ffb700 1 -- 172.21.15.78:0/2593546779 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81d80095f0 con 0x7f81dc060280 2024-01-21T08:06:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.452+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81dc108110 con 0x7f81dc060280 2024-01-21T08:06:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.452+0000 7f81d1ffb700 1 -- 172.21.15.78:0/2593546779 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f81d801cb10 con 0x7f81dc060280 2024-01-21T08:06:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:40.454 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f81c802ff60 msgr2=0x7f81c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f81c802ff60 0x7f81c8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f81cc0096a0 tx=0x7f81cc006b40).stop 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 msgr2=0x7f81dc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc1066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f81d8013510 tx=0x7f81d800b040).stop 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 shutdown_connections 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f81c802ff60 0x7f81c8032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 --2- 172.21.15.78:0/2593546779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81dc060280 0x7f81dc1066e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 >> 172.21.15.78:0/2593546779 conn(0x7f81dc0f0b70 msgr2=0x7f81dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 shutdown_connections 2024-01-21T08:06:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:40.454+0000 7f81e37e7700 1 -- 172.21.15.78:0/2593546779 wait complete. 2024-01-21T08:06:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:38.600971+0000 mgr.smithi046.lmxnwc (mgr.14164) 44 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:40.453435+0000 mon.smithi046 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.78:0/2593546779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:41.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:41.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:40.601272+0000 mgr.smithi046.lmxnwc (mgr.14164) 45 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:43.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.077+0000 7f07f53d5700 1 -- 172.21.15.78:0/236222286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 msgr2=0x7f07f00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:43.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.077+0000 7f07f53d5700 1 --2- 172.21.15.78:0/236222286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f00f66c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f07e4009fb0 tx=0x7f07e4009b30).stop 2024-01-21T08:06:43.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.077+0000 7f07f53d5700 1 -- 172.21.15.78:0/236222286 shutdown_connections 2024-01-21T08:06:43.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.077+0000 7f07f53d5700 1 --2- 172.21.15.78:0/236222286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f00f66c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:43.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.077+0000 7f07f53d5700 1 -- 172.21.15.78:0/236222286 >> 172.21.15.78:0/236222286 conn(0x7f07f00f1aa0 msgr2=0x7f07f00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:43.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.077+0000 7f07f53d5700 1 -- 172.21.15.78:0/236222286 shutdown_connections 2024-01-21T08:06:43.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.077+0000 7f07f53d5700 1 -- 172.21.15.78:0/236222286 wait complete. 2024-01-21T08:06:43.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.078+0000 7f07f53d5700 1 Processor -- start 2024-01-21T08:06:43.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.078+0000 7f07f53d5700 1 -- start start 2024-01-21T08:06:43.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.078+0000 7f07f53d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:43.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.078+0000 7f07f53d5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f01069a0 con 0x7f07f00f62a0 2024-01-21T08:06:43.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.078+0000 7f07effff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.078+0000 7f07effff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34626/0 (socket says 172.21.15.78:34626) 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.078+0000 7f07effff700 1 -- 172.21.15.78:0/1861799017 learned_addr learned my addr 172.21.15.78:0/1861799017 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.079+0000 7f07effff700 1 -- 172.21.15.78:0/1861799017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e4005040 con 0x7f07f00f62a0 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.079+0000 7f07effff700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f01064b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f07e4009830 tx=0x7f07e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.079+0000 7f07ecff9700 1 -- 172.21.15.78:0/1861799017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e4009320 con 0x7f07f00f62a0 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.079+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f0106ba0 con 0x7f07f00f62a0 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.079+0000 7f07ecff9700 1 -- 172.21.15.78:0/1861799017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07e40136c0 con 0x7f07f00f62a0 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.079+0000 7f07ecff9700 1 -- 172.21.15.78:0/1861799017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e401ee90 con 0x7f07f00f62a0 2024-01-21T08:06:43.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.079+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f0107030 con 0x7f07f00f62a0 2024-01-21T08:06:43.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.080+0000 7f07ecff9700 1 -- 172.21.15.78:0/1861799017 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07e4014440 con 0x7f07f00f62a0 2024-01-21T08:06:43.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.080+0000 7f07ecff9700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07d802ff10 0x7f07d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:43.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.080+0000 7f07ecff9700 1 -- 172.21.15.78:0/1861799017 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07e4040f40 con 0x7f07f00f62a0 2024-01-21T08:06:43.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.080+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f0045ad0 con 0x7f07f00f62a0 2024-01-21T08:06:43.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.080+0000 7f07ef7fe700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07d802ff10 0x7f07d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:43.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.081+0000 7f07ef7fe700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07d802ff10 0x7f07d80323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f07e00096a0 tx=0x7f07e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:43.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.085+0000 7f07ecff9700 1 -- 172.21.15.78:0/1861799017 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07e401c910 con 0x7f07f00f62a0 2024-01-21T08:06:43.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.289+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07f0107d50 con 0x7f07f00f62a0 2024-01-21T08:06:43.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.289+0000 7f07ecff9700 1 -- 172.21.15.78:0/1861799017 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07e401c320 con 0x7f07f00f62a0 2024-01-21T08:06:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:43.291 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07d802ff10 msgr2=0x7f07d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07d802ff10 0x7f07d80323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f07e00096a0 tx=0x7f07e0006b40).stop 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 msgr2=0x7f07f01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f01064b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f07e4009830 tx=0x7f07e400b040).stop 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 shutdown_connections 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07d802ff10 0x7f07d80323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 --2- 172.21.15.78:0/1861799017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f00f62a0 0x7f07f01064b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 >> 172.21.15.78:0/1861799017 conn(0x7f07f00f1aa0 msgr2=0x7f07f00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 shutdown_connections 2024-01-21T08:06:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:43.291+0000 7f07f53d5700 1 -- 172.21.15.78:0/1861799017 wait complete. 2024-01-21T08:06:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:43.290516+0000 mon.smithi046 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.78:0/1861799017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:44.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:44.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:44.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:42.601676+0000 mgr.smithi046.lmxnwc (mgr.14164) 46 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 -- 172.21.15.78:0/2015411236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 msgr2=0x7f2b940f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 --2- 172.21.15.78:0/2015411236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b940f66c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f2b88009fb0 tx=0x7f2b88009b30).stop 2024-01-21T08:06:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 -- 172.21.15.78:0/2015411236 shutdown_connections 2024-01-21T08:06:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 --2- 172.21.15.78:0/2015411236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b940f66c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 -- 172.21.15.78:0/2015411236 >> 172.21.15.78:0/2015411236 conn(0x7f2b940f1ac0 msgr2=0x7f2b940f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 -- 172.21.15.78:0/2015411236 shutdown_connections 2024-01-21T08:06:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 -- 172.21.15.78:0/2015411236 wait complete. 2024-01-21T08:06:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 Processor -- start 2024-01-21T08:06:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 -- start start 2024-01-21T08:06:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b94106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.047+0000 7f2b99be1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b941069f0 con 0x7f2b940f62a0 2024-01-21T08:06:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b98bdf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b94106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b98bdf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b94106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34638/0 (socket says 172.21.15.78:34638) 2024-01-21T08:06:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b98bdf700 1 -- 172.21.15.78:0/3768394949 learned_addr learned my addr 172.21.15.78:0/3768394949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b98bdf700 1 -- 172.21.15.78:0/3768394949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88005040 con 0x7f2b940f62a0 2024-01-21T08:06:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b98bdf700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b94106500 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2b88013510 tx=0x7f2b8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b917fa700 1 -- 172.21.15.78:0/3768394949 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b88009320 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b94106bf0 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b917fa700 1 -- 172.21.15.78:0/3768394949 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b88013f00 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b917fa700 1 -- 172.21.15.78:0/3768394949 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b88014dc0 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.048+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b94107080 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.049+0000 7f2b917fa700 1 -- 172.21.15.78:0/3768394949 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b88014420 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.049+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b94045ad0 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.049+0000 7f2b917fa700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f2b8402ffb0 0x7f2b84032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.049+0000 7f2b917fa700 1 -- 172.21.15.78:0/3768394949 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b88041110 con 0x7f2b940f62a0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.050+0000 7f2b93fff700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f2b8402ffb0 0x7f2b84032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:46.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.050+0000 7f2b93fff700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f2b8402ffb0 0x7f2b84032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2b800096a0 tx=0x7f2b80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.054+0000 7f2b917fa700 1 -- 172.21.15.78:0/3768394949 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b880095f0 con 0x7f2b940f62a0 2024-01-21T08:06:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.259+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b94107f30 con 0x7f2b940f62a0 2024-01-21T08:06:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.260+0000 7f2b917fa700 1 -- 172.21.15.78:0/3768394949 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b8801cb10 con 0x7f2b940f62a0 2024-01-21T08:06:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:46.263 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:46.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f2b8402ffb0 msgr2=0x7f2b84032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:46.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f2b8402ffb0 0x7f2b84032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2b800096a0 tx=0x7f2b80006b40).stop 2024-01-21T08:06:46.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 msgr2=0x7f2b94106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:46.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b94106500 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2b88013510 tx=0x7f2b8800b040).stop 2024-01-21T08:06:46.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 shutdown_connections 2024-01-21T08:06:46.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f2b8402ffb0 0x7f2b84032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:46.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 --2- 172.21.15.78:0/3768394949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b940f62a0 0x7f2b94106500 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:46.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 >> 172.21.15.78:0/3768394949 conn(0x7f2b940f1ac0 msgr2=0x7f2b940f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:46.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 shutdown_connections 2024-01-21T08:06:46.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:46.261+0000 7f2b99be1700 1 -- 172.21.15.78:0/3768394949 wait complete. 2024-01-21T08:06:46.265 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:46.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:44.601964+0000 mgr.smithi046.lmxnwc (mgr.14164) 47 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:46.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:46.261568+0000 mon.smithi046 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.78:0/3768394949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:47.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:47.668 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:46.602318+0000 mgr.smithi046.lmxnwc (mgr.14164) 48 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.930+0000 7ffae1cf3700 1 -- 172.21.15.78:0/1867944802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 msgr2=0x7ffadc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.930+0000 7ffae1cf3700 1 --2- 172.21.15.78:0/1867944802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc0f90f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7ffac4009fb0 tx=0x7ffac4009b30).stop 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.930+0000 7ffae1cf3700 1 -- 172.21.15.78:0/1867944802 shutdown_connections 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.930+0000 7ffae1cf3700 1 --2- 172.21.15.78:0/1867944802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc0f90f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.930+0000 7ffae1cf3700 1 -- 172.21.15.78:0/1867944802 >> 172.21.15.78:0/1867944802 conn(0x7ffadc0f0b70 msgr2=0x7ffadc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.930+0000 7ffae1cf3700 1 -- 172.21.15.78:0/1867944802 shutdown_connections 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.930+0000 7ffae1cf3700 1 -- 172.21.15.78:0/1867944802 wait complete. 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffae1cf3700 1 Processor -- start 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffae1cf3700 1 -- start start 2024-01-21T08:06:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffae1cf3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffae1cf3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc106bd0 con 0x7ffadc060280 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffadb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffadb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34656/0 (socket says 172.21.15.78:34656) 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffadb7fe700 1 -- 172.21.15.78:0/3010115649 learned_addr learned my addr 172.21.15.78:0/3010115649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.931+0000 7ffadb7fe700 1 -- 172.21.15.78:0/3010115649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4005040 con 0x7ffadc060280 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.932+0000 7ffadb7fe700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc1066e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ffac4013510 tx=0x7ffac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.932+0000 7ffad3fff700 1 -- 172.21.15.78:0/3010115649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffac4009320 con 0x7ffadc060280 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.932+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc106dd0 con 0x7ffadc060280 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.932+0000 7ffad3fff700 1 -- 172.21.15.78:0/3010115649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffac4013f00 con 0x7ffadc060280 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.932+0000 7ffad3fff700 1 -- 172.21.15.78:0/3010115649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffac4014dc0 con 0x7ffadc060280 2024-01-21T08:06:48.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.932+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc107260 con 0x7ffadc060280 2024-01-21T08:06:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.932+0000 7ffad3fff700 1 -- 172.21.15.78:0/3010115649 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffac401d3d0 con 0x7ffadc060280 2024-01-21T08:06:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.933+0000 7ffad3fff700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ffac802ff60 0x7ffac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.933+0000 7ffad3fff700 1 -- 172.21.15.78:0/3010115649 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffac4041120 con 0x7ffadc060280 2024-01-21T08:06:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.933+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc045ad0 con 0x7ffadc060280 2024-01-21T08:06:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.933+0000 7ffadaffd700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ffac802ff60 0x7ffac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.933+0000 7ffadaffd700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ffac802ff60 0x7ffac8032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ffacc0096a0 tx=0x7ffacc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:48.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:48.938+0000 7ffad3fff700 1 -- 172.21.15.78:0/3010115649 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffac4012920 con 0x7ffadc060280 2024-01-21T08:06:49.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.142+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffadc107f80 con 0x7ffadc060280 2024-01-21T08:06:49.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.144+0000 7ffad3fff700 1 -- 172.21.15.78:0/3010115649 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffac402b020 con 0x7ffadc060280 2024-01-21T08:06:49.146 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:49.146 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:49.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ffac802ff60 msgr2=0x7ffac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ffac802ff60 0x7ffac8032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ffacc0096a0 tx=0x7ffacc006b40).stop 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 msgr2=0x7ffadc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc1066e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ffac4013510 tx=0x7ffac400b040).stop 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 shutdown_connections 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ffac802ff60 0x7ffac8032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 --2- 172.21.15.78:0/3010115649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffadc060280 0x7ffadc1066e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 >> 172.21.15.78:0/3010115649 conn(0x7ffadc0f0b70 msgr2=0x7ffadc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 shutdown_connections 2024-01-21T08:06:49.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:49.145+0000 7ffae1cf3700 1 -- 172.21.15.78:0/3010115649 wait complete. 2024-01-21T08:06:49.149 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:49.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:49.144982+0000 mon.smithi046 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.78:0/3010115649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:50.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:50.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:48.602713+0000 mgr.smithi046.lmxnwc (mgr.14164) 49 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:50.603131+0000 mgr.smithi046.lmxnwc (mgr.14164) 50 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:51.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.762+0000 7f1f14506700 1 -- 172.21.15.78:0/476499021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 msgr2=0x7f1f0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.762+0000 7f1f14506700 1 --2- 172.21.15.78:0/476499021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c0f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1f04009960 tx=0x7f1f04004d10).stop 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.762+0000 7f1f14506700 1 -- 172.21.15.78:0/476499021 shutdown_connections 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.762+0000 7f1f14506700 1 --2- 172.21.15.78:0/476499021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c0f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.762+0000 7f1f14506700 1 -- 172.21.15.78:0/476499021 >> 172.21.15.78:0/476499021 conn(0x7f1f0c0f0b70 msgr2=0x7f1f0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.762+0000 7f1f14506700 1 -- 172.21.15.78:0/476499021 shutdown_connections 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.762+0000 7f1f14506700 1 -- 172.21.15.78:0/476499021 wait complete. 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f14506700 1 Processor -- start 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f14506700 1 -- start start 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f14506700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f14506700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c105b20 con 0x7f1f0c0f5410 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f122a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:51.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f122a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34666/0 (socket says 172.21.15.78:34666) 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f122a2700 1 -- 172.21.15.78:0/1620686525 learned_addr learned my addr 172.21.15.78:0/1620686525 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f122a2700 1 -- 172.21.15.78:0/1620686525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f04009640 con 0x7f1f0c0f5410 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f122a2700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c105630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1f04004b70 tx=0x7f1f04004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.763+0000 7f1f0affd700 1 -- 172.21.15.78:0/1620686525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f04018070 con 0x7f1f0c0f5410 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.764+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0c105d20 con 0x7f1f0c0f5410 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.764+0000 7f1f0affd700 1 -- 172.21.15.78:0/1620686525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f0400b870 con 0x7f1f0c0f5410 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.764+0000 7f1f0affd700 1 -- 172.21.15.78:0/1620686525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f0401c560 con 0x7f1f0c0f5410 2024-01-21T08:06:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.764+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0c1061b0 con 0x7f1f0c0f5410 2024-01-21T08:06:51.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.764+0000 7f1f0affd700 1 -- 172.21.15.78:0/1620686525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f04013630 con 0x7f1f0c0f5410 2024-01-21T08:06:51.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.764+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0c045ad0 con 0x7f1f0c0f5410 2024-01-21T08:06:51.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.764+0000 7f1f0affd700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ef402ff60 0x7f1ef4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:51.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.765+0000 7f1f0affd700 1 -- 172.21.15.78:0/1620686525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f040411d0 con 0x7f1f0c0f5410 2024-01-21T08:06:51.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.765+0000 7f1f11aa1700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ef402ff60 0x7f1ef4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:51.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.765+0000 7f1f11aa1700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ef402ff60 0x7f1ef4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1f000096a0 tx=0x7f1f00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:51.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.769+0000 7f1f0affd700 1 -- 172.21.15.78:0/1620686525 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f04004810 con 0x7f1f0c0f5410 2024-01-21T08:06:51.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.975+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f0c0fae60 con 0x7f1f0c0f5410 2024-01-21T08:06:51.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.975+0000 7f1f0affd700 1 -- 172.21.15.78:0/1620686525 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f04012380 con 0x7f1f0c0f5410 2024-01-21T08:06:51.977 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:51.977 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ef402ff60 msgr2=0x7f1ef4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ef402ff60 0x7f1ef4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1f000096a0 tx=0x7f1f00006b40).stop 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 msgr2=0x7f1f0c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c105630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1f04004b70 tx=0x7f1f04004d10).stop 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 shutdown_connections 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1ef402ff60 0x7f1ef4032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 --2- 172.21.15.78:0/1620686525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f0c0f5410 0x7f1f0c105630 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 >> 172.21.15.78:0/1620686525 conn(0x7f1f0c0f0b70 msgr2=0x7f1f0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 shutdown_connections 2024-01-21T08:06:51.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:51.977+0000 7f1f14506700 1 -- 172.21.15.78:0/1620686525 wait complete. 2024-01-21T08:06:51.980 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:52.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:51.976511+0000 mon.smithi046 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.78:0/1620686525' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:53.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:53.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:52.603528+0000 mgr.smithi046.lmxnwc (mgr.14164) 51 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:54.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.667+0000 7f12cb3f1700 1 -- 172.21.15.78:0/148799061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 msgr2=0x7f12c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:54.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.667+0000 7f12cb3f1700 1 --2- 172.21.15.78:0/148799061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c40f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f12b4009fb0 tx=0x7f12b4009b30).stop 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.667+0000 7f12cb3f1700 1 -- 172.21.15.78:0/148799061 shutdown_connections 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.667+0000 7f12cb3f1700 1 --2- 172.21.15.78:0/148799061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c40f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.667+0000 7f12cb3f1700 1 -- 172.21.15.78:0/148799061 >> 172.21.15.78:0/148799061 conn(0x7f12c40f0b70 msgr2=0x7f12c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.667+0000 7f12cb3f1700 1 -- 172.21.15.78:0/148799061 shutdown_connections 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.667+0000 7f12cb3f1700 1 -- 172.21.15.78:0/148799061 wait complete. 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12cb3f1700 1 Processor -- start 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12cb3f1700 1 -- start start 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12cb3f1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12cb3f1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c4106c00 con 0x7f12c40f5410 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12c918d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12c918d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:41320/0 (socket says 172.21.15.78:41320) 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12c918d700 1 -- 172.21.15.78:0/3497768773 learned_addr learned my addr 172.21.15.78:0/3497768773 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12c918d700 1 -- 172.21.15.78:0/3497768773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4005040 con 0x7f12c40f5410 2024-01-21T08:06:54.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12c918d700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c4106710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f12b4013510 tx=0x7f12b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:54.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.668+0000 7f12b9ffb700 1 -- 172.21.15.78:0/3497768773 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12b4009320 con 0x7f12c40f5410 2024-01-21T08:06:54.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.669+0000 7f12b9ffb700 1 -- 172.21.15.78:0/3497768773 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12b4013f00 con 0x7f12c40f5410 2024-01-21T08:06:54.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.669+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c4106e00 con 0x7f12c40f5410 2024-01-21T08:06:54.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.669+0000 7f12b9ffb700 1 -- 172.21.15.78:0/3497768773 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12b4014d90 con 0x7f12c40f5410 2024-01-21T08:06:54.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.669+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c4107290 con 0x7f12c40f5410 2024-01-21T08:06:54.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.669+0000 7f12b9ffb700 1 -- 172.21.15.78:0/3497768773 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12b40095f0 con 0x7f12c40f5410 2024-01-21T08:06:54.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.670+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c4045ad0 con 0x7f12c40f5410 2024-01-21T08:06:54.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.670+0000 7f12b9ffb700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12b002ff60 0x7f12b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:54.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.670+0000 7f12b9ffb700 1 -- 172.21.15.78:0/3497768773 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12b4040fe0 con 0x7f12c40f5410 2024-01-21T08:06:54.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.670+0000 7f12c898c700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12b002ff60 0x7f12b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:54.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.670+0000 7f12c898c700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12b002ff60 0x7f12b0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f12c00096a0 tx=0x7f12c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:54.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.674+0000 7f12b9ffb700 1 -- 172.21.15.78:0/3497768773 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12b401d7f0 con 0x7f12c40f5410 2024-01-21T08:06:54.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.877+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12c40fae60 con 0x7f12c40f5410 2024-01-21T08:06:54.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.878+0000 7f12b9ffb700 1 -- 172.21.15.78:0/3497768773 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12b4010230 con 0x7f12c40f5410 2024-01-21T08:06:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:54.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12b002ff60 msgr2=0x7f12b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:54.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12b002ff60 0x7f12b0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f12c00096a0 tx=0x7f12c0006b40).stop 2024-01-21T08:06:54.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 msgr2=0x7f12c4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:54.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c4106710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f12b4013510 tx=0x7f12b400b040).stop 2024-01-21T08:06:54.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 shutdown_connections 2024-01-21T08:06:54.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12b002ff60 0x7f12b0032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:54.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 --2- 172.21.15.78:0/3497768773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c40f5410 0x7f12c4106710 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:54.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 >> 172.21.15.78:0/3497768773 conn(0x7f12c40f0b70 msgr2=0x7f12c405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:54.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 shutdown_connections 2024-01-21T08:06:54.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:54.879+0000 7f12cb3f1700 1 -- 172.21.15.78:0/3497768773 wait complete. 2024-01-21T08:06:54.882 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:54.878654+0000 mon.smithi046 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.78:0/3497768773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:56.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:56.312 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:06:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:54.603900+0000 mgr.smithi046.lmxnwc (mgr.14164) 52 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:57.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.555+0000 7f9e534c8700 1 -- 172.21.15.78:0/1300922679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f4740 msgr2=0x7f9e4c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.555+0000 7f9e534c8700 1 --2- 172.21.15.78:0/1300922679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f4740 0x7f9e4c0f4b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f9e3c009fb0 tx=0x7f9e3c009b30).stop 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.555+0000 7f9e534c8700 1 -- 172.21.15.78:0/1300922679 shutdown_connections 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.555+0000 7f9e534c8700 1 --2- 172.21.15.78:0/1300922679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f4740 0x7f9e4c0f4b60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.555+0000 7f9e534c8700 1 -- 172.21.15.78:0/1300922679 >> 172.21.15.78:0/1300922679 conn(0x7f9e4c0f0260 msgr2=0x7f9e4c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.555+0000 7f9e534c8700 1 -- 172.21.15.78:0/1300922679 shutdown_connections 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.555+0000 7f9e534c8700 1 -- 172.21.15.78:0/1300922679 wait complete. 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e534c8700 1 Processor -- start 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e534c8700 1 -- start start 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e534c8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f72b0 0x7f9e4c0f5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:57.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e534c8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c0f76d0 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e51264700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f72b0 0x7f9e4c0f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e51264700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f72b0 0x7f9e4c0f5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:41344/0 (socket says 172.21.15.78:41344) 2024-01-21T08:06:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e51264700 1 -- 172.21.15.78:0/1465745549 learned_addr learned my addr 172.21.15.78:0/1465745549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:06:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e51264700 1 -- 172.21.15.78:0/1465745549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c005040 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.556+0000 7f9e51264700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f72b0 0x7f9e4c0f5770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9e3c009f80 tx=0x7f9e3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.557+0000 7f9e41ffb700 1 -- 172.21.15.78:0/1465745549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e3c0136c0 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.557+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c0f5c60 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.557+0000 7f9e41ffb700 1 -- 172.21.15.78:0/1465745549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e3c0095f0 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.557+0000 7f9e41ffb700 1 -- 172.21.15.78:0/1465745549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e3c014420 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.557+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c0f6150 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.558+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c059100 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.558+0000 7f9e41ffb700 1 -- 172.21.15.78:0/1465745549 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e3c025060 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.558+0000 7f9e41ffb700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9e3802ff60 0x7f9e38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.558+0000 7f9e41ffb700 1 -- 172.21.15.78:0/1465745549 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e3c040d80 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.558+0000 7f9e50a63700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9e3802ff60 0x7f9e38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:06:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.559+0000 7f9e50a63700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9e3802ff60 0x7f9e38032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f9e480096a0 tx=0x7f9e48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:06:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.562+0000 7f9e41ffb700 1 -- 172.21.15.78:0/1465745549 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e3c01cb60 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.765+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e4c0f6540 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.766+0000 7f9e41ffb700 1 -- 172.21.15.78:0/1465745549 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e3c042070 con 0x7f9e4c0f72b0 2024-01-21T08:06:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:06:57.768 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:06:57.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9e3802ff60 msgr2=0x7f9e38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9e3802ff60 0x7f9e38032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f9e480096a0 tx=0x7f9e48006b40).stop 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f72b0 msgr2=0x7f9e4c0f5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f72b0 0x7f9e4c0f5770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9e3c009f80 tx=0x7f9e3c009b30).stop 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 shutdown_connections 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9e3802ff60 0x7f9e38032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 --2- 172.21.15.78:0/1465745549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e4c0f72b0 0x7f9e4c0f5770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 >> 172.21.15.78:0/1465745549 conn(0x7f9e4c0f0260 msgr2=0x7f9e4c0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.767+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 shutdown_connections 2024-01-21T08:06:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:06:57.768+0000 7f9e534c8700 1 -- 172.21.15.78:0/1465745549 wait complete. 2024-01-21T08:06:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:06:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:56.604209+0000 mgr.smithi046.lmxnwc (mgr.14164) 53 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:06:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:06:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:06:57.766835+0000 mon.smithi046 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.78:0/1465745549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:06:59.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:06:59.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:00.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.417+0000 7faf9251d700 1 -- 172.21.15.78:0/3883334310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 msgr2=0x7faf8c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.417+0000 7faf9251d700 1 --2- 172.21.15.78:0/3883334310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c0f7d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7faf80009fb0 tx=0x7faf80009b30).stop 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.417+0000 7faf9251d700 1 -- 172.21.15.78:0/3883334310 shutdown_connections 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.417+0000 7faf9251d700 1 --2- 172.21.15.78:0/3883334310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c0f7d00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.417+0000 7faf9251d700 1 -- 172.21.15.78:0/3883334310 >> 172.21.15.78:0/3883334310 conn(0x7faf8c0f0120 msgr2=0x7faf8c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9251d700 1 -- 172.21.15.78:0/3883334310 shutdown_connections 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9251d700 1 -- 172.21.15.78:0/3883334310 wait complete. 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9251d700 1 Processor -- start 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9251d700 1 -- start start 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9251d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9251d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c1069d0 con 0x7faf8c0f78e0 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9151b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9151b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:41360/0 (socket says 172.21.15.78:41360) 2024-01-21T08:07:00.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.418+0000 7faf9151b700 1 -- 172.21.15.78:0/449553776 learned_addr learned my addr 172.21.15.78:0/449553776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.419+0000 7faf9151b700 1 -- 172.21.15.78:0/449553776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80005040 con 0x7faf8c0f78e0 2024-01-21T08:07:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.419+0000 7faf9151b700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7faf80013510 tx=0x7faf8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.419+0000 7faf7dffb700 1 -- 172.21.15.78:0/449553776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf80009320 con 0x7faf8c0f78e0 2024-01-21T08:07:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.419+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c106bd0 con 0x7faf8c0f78e0 2024-01-21T08:07:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.419+0000 7faf7dffb700 1 -- 172.21.15.78:0/449553776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf80013f00 con 0x7faf8c0f78e0 2024-01-21T08:07:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.419+0000 7faf7dffb700 1 -- 172.21.15.78:0/449553776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf80014dc0 con 0x7faf8c0f78e0 2024-01-21T08:07:00.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.419+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c107060 con 0x7faf8c0f78e0 2024-01-21T08:07:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.420+0000 7faf7dffb700 1 -- 172.21.15.78:0/449553776 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf800095f0 con 0x7faf8c0f78e0 2024-01-21T08:07:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.420+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c045ad0 con 0x7faf8c0f78e0 2024-01-21T08:07:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.420+0000 7faf7dffb700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7faf7802ff60 0x7faf78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.420+0000 7faf7dffb700 1 -- 172.21.15.78:0/449553776 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf800410e0 con 0x7faf8c0f78e0 2024-01-21T08:07:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.420+0000 7faf90d1a700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7faf7802ff60 0x7faf78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:00.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.421+0000 7faf90d1a700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7faf7802ff60 0x7faf78032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7faf880096a0 tx=0x7faf88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:00.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.425+0000 7faf7dffb700 1 -- 172.21.15.78:0/449553776 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf80014420 con 0x7faf8c0f78e0 2024-01-21T08:07:00.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.632+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf8c107eb0 con 0x7faf8c0f78e0 2024-01-21T08:07:00.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.633+0000 7faf7dffb700 1 -- 172.21.15.78:0/449553776 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf8001c920 con 0x7faf8c0f78e0 2024-01-21T08:07:00.634 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:00.635 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7faf7802ff60 msgr2=0x7faf78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7faf7802ff60 0x7faf78032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7faf880096a0 tx=0x7faf88006b40).stop 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 msgr2=0x7faf8c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7faf80013510 tx=0x7faf8000b040).stop 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 shutdown_connections 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7faf7802ff60 0x7faf78032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 --2- 172.21.15.78:0/449553776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 >> 172.21.15.78:0/449553776 conn(0x7faf8c0f0120 msgr2=0x7faf8c0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 shutdown_connections 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:00.634+0000 7faf9251d700 1 -- 172.21.15.78:0/449553776 wait complete. 2024-01-21T08:07:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:06:58.604580+0000 mgr.smithi046.lmxnwc (mgr.14164) 54 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:00.633658+0000 mon.smithi046 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.78:0/449553776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:02.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:02.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:00.604992+0000 mgr.smithi046.lmxnwc (mgr.14164) 55 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:03.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.225+0000 7f87d6e51700 1 -- 172.21.15.78:0/2914353055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 msgr2=0x7f87d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:03.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.225+0000 7f87d6e51700 1 --2- 172.21.15.78:0/2914353055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d00f55c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f87cc009fb0 tx=0x7f87cc009b30).stop 2024-01-21T08:07:03.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.225+0000 7f87d6e51700 1 -- 172.21.15.78:0/2914353055 shutdown_connections 2024-01-21T08:07:03.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.225+0000 7f87d6e51700 1 --2- 172.21.15.78:0/2914353055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d00f55c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:03.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.225+0000 7f87d6e51700 1 -- 172.21.15.78:0/2914353055 >> 172.21.15.78:0/2914353055 conn(0x7f87d00f09a0 msgr2=0x7f87d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:03.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.225+0000 7f87d6e51700 1 -- 172.21.15.78:0/2914353055 shutdown_connections 2024-01-21T08:07:03.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.225+0000 7f87d6e51700 1 -- 172.21.15.78:0/2914353055 wait complete. 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d6e51700 1 Processor -- start 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d6e51700 1 -- start start 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d6e51700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d010b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d6e51700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d0106f60 con 0x7f87d00f51a0 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d5e4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d010b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d5e4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d010b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34210/0 (socket says 172.21.15.78:34210) 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d5e4f700 1 -- 172.21.15.78:0/1244944513 learned_addr learned my addr 172.21.15.78:0/1244944513 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.226+0000 7f87d5e4f700 1 -- 172.21.15.78:0/1244944513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc005040 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.227+0000 7f87d5e4f700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d010b570 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f87cc013510 tx=0x7f87cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.227+0000 7f87c67fc700 1 -- 172.21.15.78:0/1244944513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87cc009320 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.227+0000 7f87d6e51700 1 -- 172.21.15.78:0/1244944513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d0060280 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.227+0000 7f87d6e51700 1 -- 172.21.15.78:0/1244944513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d010ba60 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.227+0000 7f87c67fc700 1 -- 172.21.15.78:0/1244944513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87cc013f00 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.227+0000 7f87c67fc700 1 -- 172.21.15.78:0/1244944513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87cc014d90 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.228+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d0045ad0 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.228+0000 7f87c67fc700 1 -- 172.21.15.78:0/1244944513 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87cc0095f0 con 0x7f87d00f51a0 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.228+0000 7f87c67fc700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f87bc02ff60 0x7f87bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.228+0000 7f87c67fc700 1 -- 172.21.15.78:0/1244944513 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87cc040ff0 con 0x7f87d00f51a0 2024-01-21T08:07:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.228+0000 7f87d564e700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f87bc02ff60 0x7f87bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.229+0000 7f87d564e700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f87bc02ff60 0x7f87bc032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f87c00096a0 tx=0x7f87c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.233+0000 7f87c67fc700 1 -- 172.21.15.78:0/1244944513 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87cc014420 con 0x7f87d00f51a0 2024-01-21T08:07:03.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.437+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87d00fabe0 con 0x7f87d00f51a0 2024-01-21T08:07:03.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.437+0000 7f87c67fc700 1 -- 172.21.15.78:0/1244944513 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87cc012980 con 0x7f87d00f51a0 2024-01-21T08:07:03.439 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:03.440 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:03.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f87bc02ff60 msgr2=0x7f87bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:03.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f87bc02ff60 0x7f87bc032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f87c00096a0 tx=0x7f87c0006b40).stop 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 msgr2=0x7f87d010b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d010b570 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f87cc013510 tx=0x7f87cc00b040).stop 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 shutdown_connections 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f87bc02ff60 0x7f87bc032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 --2- 172.21.15.78:0/1244944513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d00f51a0 0x7f87d010b570 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 >> 172.21.15.78:0/1244944513 conn(0x7f87d00f09a0 msgr2=0x7f87d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 shutdown_connections 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:03.439+0000 7f87b3fff700 1 -- 172.21.15.78:0/1244944513 wait complete. 2024-01-21T08:07:03.442 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:04.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:02.605356+0000 mgr.smithi046.lmxnwc (mgr.14164) 56 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:04.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:03.438544+0000 mon.smithi046 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.78:0/1244944513' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:04.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:04.838 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.099+0000 7fc415f3d700 1 -- 172.21.15.78:0/2969963001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 msgr2=0x7fc4100f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.099+0000 7fc415f3d700 1 --2- 172.21.15.78:0/2969963001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4100f9150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc400009fb0 tx=0x7fc400009b30).stop 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.099+0000 7fc415f3d700 1 -- 172.21.15.78:0/2969963001 shutdown_connections 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.099+0000 7fc415f3d700 1 --2- 172.21.15.78:0/2969963001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4100f9150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.099+0000 7fc415f3d700 1 -- 172.21.15.78:0/2969963001 >> 172.21.15.78:0/2969963001 conn(0x7fc4100f0b70 msgr2=0x7fc4100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.099+0000 7fc415f3d700 1 -- 172.21.15.78:0/2969963001 shutdown_connections 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.099+0000 7fc415f3d700 1 -- 172.21.15.78:0/2969963001 wait complete. 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.100+0000 7fc415f3d700 1 Processor -- start 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.100+0000 7fc415f3d700 1 -- start start 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.100+0000 7fc415f3d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4101066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.100+0000 7fc415f3d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc410106ba0 con 0x7fc410060220 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.100+0000 7fc40f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4101066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.100+0000 7fc40f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4101066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34230/0 (socket says 172.21.15.78:34230) 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.100+0000 7fc40f7fe700 1 -- 172.21.15.78:0/1646523285 learned_addr learned my addr 172.21.15.78:0/1646523285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.101+0000 7fc40f7fe700 1 -- 172.21.15.78:0/1646523285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400005040 con 0x7fc410060220 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.101+0000 7fc40f7fe700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4101066b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc400013510 tx=0x7fc40000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.101+0000 7fc3f7fff700 1 -- 172.21.15.78:0/1646523285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc400009320 con 0x7fc410060220 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.101+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc410106da0 con 0x7fc410060220 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.101+0000 7fc3f7fff700 1 -- 172.21.15.78:0/1646523285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc400013f00 con 0x7fc410060220 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.101+0000 7fc3f7fff700 1 -- 172.21.15.78:0/1646523285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc400014dc0 con 0x7fc410060220 2024-01-21T08:07:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.101+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc410107230 con 0x7fc410060220 2024-01-21T08:07:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.102+0000 7fc3f7fff700 1 -- 172.21.15.78:0/1646523285 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc400014420 con 0x7fc410060220 2024-01-21T08:07:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.102+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc410045ad0 con 0x7fc410060220 2024-01-21T08:07:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.102+0000 7fc3f7fff700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3f802ff60 0x7fc3f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.102+0000 7fc3f7fff700 1 -- 172.21.15.78:0/1646523285 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4000410d0 con 0x7fc410060220 2024-01-21T08:07:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.102+0000 7fc40effd700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3f802ff60 0x7fc3f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.103+0000 7fc40effd700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3f802ff60 0x7fc3f8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc4040096a0 tx=0x7fc404006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:06.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.107+0000 7fc3f7fff700 1 -- 172.21.15.78:0/1646523285 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4000095f0 con 0x7fc410060220 2024-01-21T08:07:06.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.312+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4101080e0 con 0x7fc410060220 2024-01-21T08:07:06.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.313+0000 7fc3f7fff700 1 -- 172.21.15.78:0/1646523285 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc40001cb10 con 0x7fc410060220 2024-01-21T08:07:06.315 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:06.315 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3f802ff60 msgr2=0x7fc3f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3f802ff60 0x7fc3f8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc4040096a0 tx=0x7fc404006b40).stop 2024-01-21T08:07:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 msgr2=0x7fc4101066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4101066b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc400013510 tx=0x7fc40000b040).stop 2024-01-21T08:07:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 shutdown_connections 2024-01-21T08:07:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3f802ff60 0x7fc3f8032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 --2- 172.21.15.78:0/1646523285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc410060220 0x7fc4101066b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 >> 172.21.15.78:0/1646523285 conn(0x7fc4100f0b70 msgr2=0x7fc4100f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 shutdown_connections 2024-01-21T08:07:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:06.314+0000 7fc415f3d700 1 -- 172.21.15.78:0/1646523285 wait complete. 2024-01-21T08:07:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:04.605747+0000 mgr.smithi046.lmxnwc (mgr.14164) 57 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:07.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:07.726 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:06.313797+0000 mon.smithi046 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.78:0/1646523285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:06.606116+0000 mgr.smithi046.lmxnwc (mgr.14164) 58 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:08.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 -- 172.21.15.78:0/148045066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 msgr2=0x7fc2e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:08.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 --2- 172.21.15.78:0/148045066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e00f66c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fc2d0009fb0 tx=0x7fc2d0009b30).stop 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 -- 172.21.15.78:0/148045066 shutdown_connections 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 --2- 172.21.15.78:0/148045066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e00f66c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 -- 172.21.15.78:0/148045066 >> 172.21.15.78:0/148045066 conn(0x7fc2e00f1ac0 msgr2=0x7fc2e00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 -- 172.21.15.78:0/148045066 shutdown_connections 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 -- 172.21.15.78:0/148045066 wait complete. 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 Processor -- start 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.972+0000 7fc2df59e700 1 -- start start 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.973+0000 7fc2df59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.973+0000 7fc2df59e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e01069f0 con 0x7fc2e00f62a0 2024-01-21T08:07:08.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.973+0000 7fc2de59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.973+0000 7fc2de59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34258/0 (socket says 172.21.15.78:34258) 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.973+0000 7fc2de59c700 1 -- 172.21.15.78:0/2707296188 learned_addr learned my addr 172.21.15.78:0/2707296188 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.973+0000 7fc2de59c700 1 -- 172.21.15.78:0/2707296188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d0005040 con 0x7fc2e00f62a0 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.973+0000 7fc2de59c700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e0106500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fc2d0013510 tx=0x7fc2d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.974+0000 7fc2d6ffd700 1 -- 172.21.15.78:0/2707296188 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d0009320 con 0x7fc2e00f62a0 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.974+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e0106bf0 con 0x7fc2e00f62a0 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.974+0000 7fc2d6ffd700 1 -- 172.21.15.78:0/2707296188 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2d0013f00 con 0x7fc2e00f62a0 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.974+0000 7fc2d6ffd700 1 -- 172.21.15.78:0/2707296188 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d0014dc0 con 0x7fc2e00f62a0 2024-01-21T08:07:08.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.974+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e0107080 con 0x7fc2e00f62a0 2024-01-21T08:07:08.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.974+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e0045ad0 con 0x7fc2e00f62a0 2024-01-21T08:07:08.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.974+0000 7fc2d6ffd700 1 -- 172.21.15.78:0/2707296188 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2d00095f0 con 0x7fc2e00f62a0 2024-01-21T08:07:08.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.975+0000 7fc2d6ffd700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc2cc02ff60 0x7fc2cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:08.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.975+0000 7fc2d6ffd700 1 -- 172.21.15.78:0/2707296188 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2d0041020 con 0x7fc2e00f62a0 2024-01-21T08:07:08.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.975+0000 7fc2ddd9b700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc2cc02ff60 0x7fc2cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:08.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.975+0000 7fc2ddd9b700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc2cc02ff60 0x7fc2cc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc2c80096a0 tx=0x7fc2c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:08.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:08.979+0000 7fc2d6ffd700 1 -- 172.21.15.78:0/2707296188 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2d0014420 con 0x7fc2e00f62a0 2024-01-21T08:07:09.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.184+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2e0107eb0 con 0x7fc2e00f62a0 2024-01-21T08:07:09.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.185+0000 7fc2d6ffd700 1 -- 172.21.15.78:0/2707296188 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2d001c920 con 0x7fc2e00f62a0 2024-01-21T08:07:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:09.188 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc2cc02ff60 msgr2=0x7fc2cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc2cc02ff60 0x7fc2cc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc2c80096a0 tx=0x7fc2c8006b40).stop 2024-01-21T08:07:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 msgr2=0x7fc2e0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e0106500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fc2d0013510 tx=0x7fc2d000b040).stop 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 shutdown_connections 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc2cc02ff60 0x7fc2cc032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 --2- 172.21.15.78:0/2707296188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2e00f62a0 0x7fc2e0106500 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 >> 172.21.15.78:0/2707296188 conn(0x7fc2e00f1ac0 msgr2=0x7fc2e00f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 shutdown_connections 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:09.187+0000 7fc2df59e700 1 -- 172.21.15.78:0/2707296188 wait complete. 2024-01-21T08:07:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:09.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:09.186274+0000 mon.smithi046 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.78:0/2707296188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:10.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:10.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:08.606471+0000 mgr.smithi046.lmxnwc (mgr.14164) 59 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:11.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.786+0000 7f5091262700 1 -- 172.21.15.78:0/569284800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 msgr2=0x7f508c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.786+0000 7f5091262700 1 --2- 172.21.15.78:0/569284800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f9150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f5084009960 tx=0x7f5084004d10).stop 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.786+0000 7f5091262700 1 -- 172.21.15.78:0/569284800 shutdown_connections 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.786+0000 7f5091262700 1 --2- 172.21.15.78:0/569284800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f9150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.786+0000 7f5091262700 1 -- 172.21.15.78:0/569284800 >> 172.21.15.78:0/569284800 conn(0x7f508c0f0b70 msgr2=0x7f508c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.786+0000 7f5091262700 1 -- 172.21.15.78:0/569284800 shutdown_connections 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.786+0000 7f5091262700 1 -- 172.21.15.78:0/569284800 wait complete. 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.787+0000 7f5091262700 1 Processor -- start 2024-01-21T08:07:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.787+0000 7f5091262700 1 -- start start 2024-01-21T08:07:11.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.787+0000 7f5091262700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:11.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.787+0000 7f5091262700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c0f5050 con 0x7f508c060220 2024-01-21T08:07:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.787+0000 7f508affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.787+0000 7f508affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:34276/0 (socket says 172.21.15.78:34276) 2024-01-21T08:07:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.787+0000 7f508affd700 1 -- 172.21.15.78:0/2033040901 learned_addr learned my addr 172.21.15.78:0/2033040901 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.788+0000 7f508affd700 1 -- 172.21.15.78:0/2033040901 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084009640 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.788+0000 7f508affd700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f6bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5084007b20 tx=0x7f5084004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.788+0000 7f50837fe700 1 -- 172.21.15.78:0/2033040901 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5084018070 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.788+0000 7f50837fe700 1 -- 172.21.15.78:0/2033040901 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5084007e60 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.788+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c0f51f0 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.788+0000 7f50837fe700 1 -- 172.21.15.78:0/2033040901 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f508401d420 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.788+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c0f5680 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.789+0000 7f50837fe700 1 -- 172.21.15.78:0/2033040901 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5084013630 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.789+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508c059100 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.789+0000 7f50837fe700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f507802ffb0 0x7f5078032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.789+0000 7f50837fe700 1 -- 172.21.15.78:0/2033040901 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5084041630 con 0x7f508c060220 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.789+0000 7f508a7fc700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f507802ffb0 0x7f5078032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.789+0000 7f508a7fc700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f507802ffb0 0x7f5078032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f50740096a0 tx=0x7f5074006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:11.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.794+0000 7f50837fe700 1 -- 172.21.15.78:0/2033040901 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5084004810 con 0x7f508c060220 2024-01-21T08:07:11.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.996+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f508c045ad0 con 0x7f508c060220 2024-01-21T08:07:11.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.997+0000 7f50837fe700 1 -- 172.21.15.78:0/2033040901 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5084011320 con 0x7f508c060220 2024-01-21T08:07:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:12.000 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.998+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f507802ffb0 msgr2=0x7f5078032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.998+0000 7f5091262700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f507802ffb0 0x7f5078032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f50740096a0 tx=0x7f5074006b40).stop 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.998+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 msgr2=0x7f508c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.998+0000 7f5091262700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f6bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5084007b20 tx=0x7f5084004d10).stop 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.998+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 shutdown_connections 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.998+0000 7f5091262700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f507802ffb0 0x7f5078032470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.998+0000 7f5091262700 1 --2- 172.21.15.78:0/2033040901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c060220 0x7f508c0f6bd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.999+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 >> 172.21.15.78:0/2033040901 conn(0x7f508c0f0b70 msgr2=0x7f508c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:12.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.999+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 shutdown_connections 2024-01-21T08:07:12.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:11.999+0000 7f5091262700 1 -- 172.21.15.78:0/2033040901 wait complete. 2024-01-21T08:07:12.002 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:10.606779+0000 mgr.smithi046.lmxnwc (mgr.14164) 60 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:12.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:11.997957+0000 mon.smithi046 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.78:0/2033040901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:13.368 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:13.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 -- 172.21.15.78:0/499603213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 msgr2=0x7f1b2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 --2- 172.21.15.78:0/499603213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c0f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f1b20009fb0 tx=0x7f1b20009b30).stop 2024-01-21T08:07:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 -- 172.21.15.78:0/499603213 shutdown_connections 2024-01-21T08:07:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 --2- 172.21.15.78:0/499603213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c0f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 -- 172.21.15.78:0/499603213 >> 172.21.15.78:0/499603213 conn(0x7f1b2c0f0b70 msgr2=0x7f1b2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 -- 172.21.15.78:0/499603213 shutdown_connections 2024-01-21T08:07:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 -- 172.21.15.78:0/499603213 wait complete. 2024-01-21T08:07:14.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 Processor -- start 2024-01-21T08:07:14.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.579+0000 7f1b32a75700 1 -- start start 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.580+0000 7f1b32a75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.580+0000 7f1b32a75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c060220 con 0x7f1b2c0f5410 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.580+0000 7f1b30811700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.580+0000 7f1b30811700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:54092/0 (socket says 172.21.15.78:54092) 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.580+0000 7f1b30811700 1 -- 172.21.15.78:0/4140543710 learned_addr learned my addr 172.21.15.78:0/4140543710 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.580+0000 7f1b30811700 1 -- 172.21.15.78:0/4140543710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b20005040 con 0x7f1b2c0f5410 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.581+0000 7f1b30811700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c109450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1b20013510 tx=0x7f1b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.581+0000 7f1b297fa700 1 -- 172.21.15.78:0/4140543710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b20009320 con 0x7f1b2c0f5410 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.581+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c060420 con 0x7f1b2c0f5410 2024-01-21T08:07:14.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.581+0000 7f1b297fa700 1 -- 172.21.15.78:0/4140543710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b20013f00 con 0x7f1b2c0f5410 2024-01-21T08:07:14.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.581+0000 7f1b297fa700 1 -- 172.21.15.78:0/4140543710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b20014d90 con 0x7f1b2c0f5410 2024-01-21T08:07:14.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.581+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c109b00 con 0x7f1b2c0f5410 2024-01-21T08:07:14.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.582+0000 7f1b297fa700 1 -- 172.21.15.78:0/4140543710 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b200095f0 con 0x7f1b2c0f5410 2024-01-21T08:07:14.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.582+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c059100 con 0x7f1b2c0f5410 2024-01-21T08:07:14.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.582+0000 7f1b297fa700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b1c02ff60 0x7f1b1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:14.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.582+0000 7f1b297fa700 1 -- 172.21.15.78:0/4140543710 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b200410c0 con 0x7f1b2c0f5410 2024-01-21T08:07:14.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.582+0000 7f1b2bfff700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b1c02ff60 0x7f1b1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:14.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.582+0000 7f1b2bfff700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b1c02ff60 0x7f1b1c032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1b180096a0 tx=0x7f1b18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:14.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.586+0000 7f1b297fa700 1 -- 172.21.15.78:0/4140543710 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b20014420 con 0x7f1b2c0f5410 2024-01-21T08:07:14.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:12.607050+0000 mgr.smithi046.lmxnwc (mgr.14164) 61 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.790+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b2c045ad0 con 0x7f1b2c0f5410 2024-01-21T08:07:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.790+0000 7f1b297fa700 1 -- 172.21.15.78:0/4140543710 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b20010230 con 0x7f1b2c0f5410 2024-01-21T08:07:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:14.792 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b1c02ff60 msgr2=0x7f1b1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b1c02ff60 0x7f1b1c032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1b180096a0 tx=0x7f1b18006b40).stop 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 msgr2=0x7f1b2c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c109450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1b20013510 tx=0x7f1b2000b040).stop 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 shutdown_connections 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b1c02ff60 0x7f1b1c032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 --2- 172.21.15.78:0/4140543710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b2c0f5410 0x7f1b2c109450 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 >> 172.21.15.78:0/4140543710 conn(0x7f1b2c0f0b70 msgr2=0x7f1b2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 shutdown_connections 2024-01-21T08:07:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:14.792+0000 7f1b32a75700 1 -- 172.21.15.78:0/4140543710 wait complete. 2024-01-21T08:07:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:14.791578+0000 mon.smithi046 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.78:0/4140543710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:16.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:16.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:14.607414+0000 mgr.smithi046.lmxnwc (mgr.14164) 62 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.443+0000 7fae2eb20700 1 -- 172.21.15.78:0/72946240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 msgr2=0x7fae280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.443+0000 7fae2eb20700 1 --2- 172.21.15.78:0/72946240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae280f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fae1c009fb0 tx=0x7fae1c009b30).stop 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.443+0000 7fae2eb20700 1 -- 172.21.15.78:0/72946240 shutdown_connections 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.443+0000 7fae2eb20700 1 --2- 172.21.15.78:0/72946240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae280f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.443+0000 7fae2eb20700 1 -- 172.21.15.78:0/72946240 >> 172.21.15.78:0/72946240 conn(0x7fae280f1c70 msgr2=0x7fae280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.443+0000 7fae2eb20700 1 -- 172.21.15.78:0/72946240 shutdown_connections 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.443+0000 7fae2eb20700 1 -- 172.21.15.78:0/72946240 wait complete. 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.444+0000 7fae2eb20700 1 Processor -- start 2024-01-21T08:07:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.444+0000 7fae2eb20700 1 -- start start 2024-01-21T08:07:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.444+0000 7fae2eb20700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae28106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.444+0000 7fae2eb20700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae28106c20 con 0x7fae280f6510 2024-01-21T08:07:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.444+0000 7fae2c8bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae28106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.444+0000 7fae2c8bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae28106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:54102/0 (socket says 172.21.15.78:54102) 2024-01-21T08:07:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae2c8bc700 1 -- 172.21.15.78:0/762887207 learned_addr learned my addr 172.21.15.78:0/762887207 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae2c8bc700 1 -- 172.21.15.78:0/762887207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c005040 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae2c8bc700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae28106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fae1c013510 tx=0x7fae1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae257fa700 1 -- 172.21.15.78:0/762887207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae1c009320 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae28106e20 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae257fa700 1 -- 172.21.15.78:0/762887207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae1c013f00 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae257fa700 1 -- 172.21.15.78:0/762887207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae1c014d90 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.445+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae281072b0 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.446+0000 7fae257fa700 1 -- 172.21.15.78:0/762887207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae1c014420 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.446+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae28045ad0 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.446+0000 7fae257fa700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fae1802ff60 0x7fae18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.446+0000 7fae257fa700 1 -- 172.21.15.78:0/762887207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae1c041070 con 0x7fae280f6510 2024-01-21T08:07:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.446+0000 7fae27fff700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fae1802ff60 0x7fae18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.447+0000 7fae27fff700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fae1802ff60 0x7fae18032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.451+0000 7fae257fa700 1 -- 172.21.15.78:0/762887207 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae1c0095f0 con 0x7fae280f6510 2024-01-21T08:07:17.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.657+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae280fbf60 con 0x7fae280f6510 2024-01-21T08:07:17.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.657+0000 7fae257fa700 1 -- 172.21.15.78:0/762887207 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae1c01db70 con 0x7fae280f6510 2024-01-21T08:07:17.659 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:17.660 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:17.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fae1802ff60 msgr2=0x7fae18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:17.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fae1802ff60 0x7fae18032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).stop 2024-01-21T08:07:17.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 msgr2=0x7fae28106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:17.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae28106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fae1c013510 tx=0x7fae1c00b040).stop 2024-01-21T08:07:17.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 shutdown_connections 2024-01-21T08:07:17.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fae1802ff60 0x7fae18032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:17.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 --2- 172.21.15.78:0/762887207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae280f6510 0x7fae28106730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:17.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 >> 172.21.15.78:0/762887207 conn(0x7fae280f1c70 msgr2=0x7fae280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:17.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 shutdown_connections 2024-01-21T08:07:17.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:17.659+0000 7fae2eb20700 1 -- 172.21.15.78:0/762887207 wait complete. 2024-01-21T08:07:17.662 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:18.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:16.607794+0000 mgr.smithi046.lmxnwc (mgr.14164) 63 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:18.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:17.658502+0000 mon.smithi046 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.78:0/762887207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:19.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:19.067 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 -- 172.21.15.78:0/3312340649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 msgr2=0x7fb6fc0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 --2- 172.21.15.78:0/3312340649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc0f7d00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb6ec009fb0 tx=0x7fb6ec009b30).stop 2024-01-21T08:07:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 -- 172.21.15.78:0/3312340649 shutdown_connections 2024-01-21T08:07:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 --2- 172.21.15.78:0/3312340649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc0f7d00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 -- 172.21.15.78:0/3312340649 >> 172.21.15.78:0/3312340649 conn(0x7fb6fc0f0120 msgr2=0x7fb6fc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 -- 172.21.15.78:0/3312340649 shutdown_connections 2024-01-21T08:07:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 -- 172.21.15.78:0/3312340649 wait complete. 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.326+0000 7fb70216e700 1 Processor -- start 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.327+0000 7fb70216e700 1 -- start start 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.327+0000 7fb70216e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.327+0000 7fb70216e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1069d0 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.327+0000 7fb70116c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.327+0000 7fb70116c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:54116/0 (socket says 172.21.15.78:54116) 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.327+0000 7fb70116c700 1 -- 172.21.15.78:0/2614916345 learned_addr learned my addr 172.21.15.78:0/2614916345 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.327+0000 7fb70116c700 1 -- 172.21.15.78:0/2614916345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec005040 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.328+0000 7fb70116c700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc1064e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb6ec013510 tx=0x7fb6ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.328+0000 7fb6f1ffb700 1 -- 172.21.15.78:0/2614916345 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6ec009320 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.328+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6fc106bd0 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.328+0000 7fb6f1ffb700 1 -- 172.21.15.78:0/2614916345 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6ec013f00 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.328+0000 7fb6f1ffb700 1 -- 172.21.15.78:0/2614916345 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6ec014dc0 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.328+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6fc107060 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.329+0000 7fb6f1ffb700 1 -- 172.21.15.78:0/2614916345 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6ec0095f0 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.329+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc045ad0 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.329+0000 7fb6f1ffb700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb6e802ff60 0x7fb6e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.329+0000 7fb6f1ffb700 1 -- 172.21.15.78:0/2614916345 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6ec041020 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.329+0000 7fb70096b700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb6e802ff60 0x7fb6e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.330+0000 7fb70096b700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb6e802ff60 0x7fb6e8032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb6f80096a0 tx=0x7fb6f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:20.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.333+0000 7fb6f1ffb700 1 -- 172.21.15.78:0/2614916345 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6ec014420 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.540+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6fc107eb0 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.541+0000 7fb6f1ffb700 1 -- 172.21.15.78:0/2614916345 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6ec01c920 con 0x7fb6fc0f78e0 2024-01-21T08:07:20.543 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:20.543 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:20.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb6e802ff60 msgr2=0x7fb6e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:20.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb6e802ff60 0x7fb6e8032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb6f80096a0 tx=0x7fb6f8006b40).stop 2024-01-21T08:07:20.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 msgr2=0x7fb6fc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:20.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc1064e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb6ec013510 tx=0x7fb6ec00b040).stop 2024-01-21T08:07:20.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 shutdown_connections 2024-01-21T08:07:20.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb6e802ff60 0x7fb6e8032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:20.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 --2- 172.21.15.78:0/2614916345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc0f78e0 0x7fb6fc1064e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:20.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 >> 172.21.15.78:0/2614916345 conn(0x7fb6fc0f0120 msgr2=0x7fb6fc0f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:20.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 shutdown_connections 2024-01-21T08:07:20.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:20.542+0000 7fb70216e700 1 -- 172.21.15.78:0/2614916345 wait complete. 2024-01-21T08:07:20.545 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:18.608134+0000 mgr.smithi046.lmxnwc (mgr.14164) 64 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:20.541710+0000 mon.smithi046 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.78:0/2614916345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:20.740134+0000 mon.smithi046 (mon.0) 162 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:07:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:20.744138+0000 mon.smithi046 (mon.0) 163 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:07:21.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:21.950 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:20.608445+0000 mgr.smithi046.lmxnwc (mgr.14164) 65 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.195+0000 7f8de0627700 1 -- 172.21.15.78:0/3135146133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 msgr2=0x7f8dd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.195+0000 7f8de0627700 1 --2- 172.21.15.78:0/3135146133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8dc8009fb0 tx=0x7f8dc8009b30).stop 2024-01-21T08:07:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.195+0000 7f8de0627700 1 -- 172.21.15.78:0/3135146133 shutdown_connections 2024-01-21T08:07:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.195+0000 7f8de0627700 1 --2- 172.21.15.78:0/3135146133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.195+0000 7f8de0627700 1 -- 172.21.15.78:0/3135146133 >> 172.21.15.78:0/3135146133 conn(0x7f8dd80f0b70 msgr2=0x7f8dd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.195+0000 7f8de0627700 1 -- 172.21.15.78:0/3135146133 shutdown_connections 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.195+0000 7f8de0627700 1 -- 172.21.15.78:0/3135146133 wait complete. 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.196+0000 7f8de0627700 1 Processor -- start 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.196+0000 7f8de0627700 1 -- start start 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.196+0000 7f8de0627700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.196+0000 7f8de0627700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80fb820 con 0x7f8dd80f5410 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.196+0000 7f8dde3c3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.196+0000 7f8dde3c3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:45606/0 (socket says 172.21.15.78:45606) 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.196+0000 7f8dde3c3700 1 -- 172.21.15.78:0/854535880 learned_addr learned my addr 172.21.15.78:0/854535880 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.197+0000 7f8dde3c3700 1 -- 172.21.15.78:0/854535880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc8005040 con 0x7f8dd80f5410 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.197+0000 7f8dde3c3700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80fd3a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8dc8013510 tx=0x7f8dc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.197+0000 7f8dceffd700 1 -- 172.21.15.78:0/854535880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc8009320 con 0x7f8dd80f5410 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.197+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd80fb9c0 con 0x7f8dd80f5410 2024-01-21T08:07:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.197+0000 7f8dceffd700 1 -- 172.21.15.78:0/854535880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dc8013f00 con 0x7f8dd80f5410 2024-01-21T08:07:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.197+0000 7f8dceffd700 1 -- 172.21.15.78:0/854535880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc8014d90 con 0x7f8dd80f5410 2024-01-21T08:07:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.197+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd80fbe50 con 0x7f8dd80f5410 2024-01-21T08:07:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.198+0000 7f8dceffd700 1 -- 172.21.15.78:0/854535880 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8dc80095f0 con 0x7f8dd80f5410 2024-01-21T08:07:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.198+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd8059100 con 0x7f8dd80f5410 2024-01-21T08:07:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.198+0000 7f8dceffd700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8dc402ff60 0x7f8dc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.198+0000 7f8dceffd700 1 -- 172.21.15.78:0/854535880 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8dc80410c0 con 0x7f8dd80f5410 2024-01-21T08:07:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.198+0000 7f8dddbc2700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8dc402ff60 0x7f8dc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.199+0000 7f8dddbc2700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8dc402ff60 0x7f8dc4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8dd40096a0 tx=0x7f8dd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:23.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.203+0000 7f8dceffd700 1 -- 172.21.15.78:0/854535880 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8dc8014420 con 0x7f8dd80f5410 2024-01-21T08:07:23.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.407+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8dd80fae60 con 0x7f8dd80f5410 2024-01-21T08:07:23.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.408+0000 7f8dceffd700 1 -- 172.21.15.78:0/854535880 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8dc801d7f0 con 0x7f8dd80f5410 2024-01-21T08:07:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:23.410 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8dc402ff60 msgr2=0x7f8dc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8dc402ff60 0x7f8dc4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8dd40096a0 tx=0x7f8dd4006b40).stop 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 msgr2=0x7f8dd80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80fd3a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8dc8013510 tx=0x7f8dc800b040).stop 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 shutdown_connections 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8dc402ff60 0x7f8dc4032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 --2- 172.21.15.78:0/854535880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd80f5410 0x7f8dd80fd3a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 >> 172.21.15.78:0/854535880 conn(0x7f8dd80f0b70 msgr2=0x7f8dd805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 shutdown_connections 2024-01-21T08:07:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:23.410+0000 7f8de0627700 1 -- 172.21.15.78:0/854535880 wait complete. 2024-01-21T08:07:23.413 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:22.608797+0000 mgr.smithi046.lmxnwc (mgr.14164) 66 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:23.408960+0000 mon.smithi046 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.78:0/854535880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:24.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:24.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:26.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.068+0000 7f85094aa700 1 -- 172.21.15.78:0/30946695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 msgr2=0x7f85040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:26.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.068+0000 7f85094aa700 1 --2- 172.21.15.78:0/30946695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f85040f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f84fc009960 tx=0x7f84fc004d10).stop 2024-01-21T08:07:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.068+0000 7f85094aa700 1 -- 172.21.15.78:0/30946695 shutdown_connections 2024-01-21T08:07:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.068+0000 7f85094aa700 1 --2- 172.21.15.78:0/30946695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f85040f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.068+0000 7f85094aa700 1 -- 172.21.15.78:0/30946695 >> 172.21.15.78:0/30946695 conn(0x7f85040f1c70 msgr2=0x7f85040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.069+0000 7f85094aa700 1 -- 172.21.15.78:0/30946695 shutdown_connections 2024-01-21T08:07:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.069+0000 7f85094aa700 1 -- 172.21.15.78:0/30946695 wait complete. 2024-01-21T08:07:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.069+0000 7f85094aa700 1 Processor -- start 2024-01-21T08:07:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.069+0000 7f85094aa700 1 -- start start 2024-01-21T08:07:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.070+0000 7f85094aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f8504106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.070+0000 7f85094aa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504106c20 con 0x7f85040f6510 2024-01-21T08:07:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.070+0000 7f8502ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f8504106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.070+0000 7f8502ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f8504106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:45622/0 (socket says 172.21.15.78:45622) 2024-01-21T08:07:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.070+0000 7f8502ffd700 1 -- 172.21.15.78:0/1483355223 learned_addr learned my addr 172.21.15.78:0/1483355223 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.070+0000 7f8502ffd700 1 -- 172.21.15.78:0/1483355223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc009640 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.070+0000 7f8502ffd700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f8504106730 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f84fc004b70 tx=0x7f84fc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.071+0000 7f84fb7fe700 1 -- 172.21.15.78:0/1483355223 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84fc018070 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.071+0000 7f84fb7fe700 1 -- 172.21.15.78:0/1483355223 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84fc00b870 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.071+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8504106e20 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.071+0000 7f84fb7fe700 1 -- 172.21.15.78:0/1483355223 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84fc01c560 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.071+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85041072b0 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.072+0000 7f84fb7fe700 1 -- 172.21.15.78:0/1483355223 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84fc01c6c0 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.072+0000 7f84fb7fe700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f84f002ff60 0x7f84f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.072+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8504045ad0 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.072+0000 7f84fb7fe700 1 -- 172.21.15.78:0/1483355223 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84fc041210 con 0x7f85040f6510 2024-01-21T08:07:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.072+0000 7f85027fc700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f84f002ff60 0x7f84f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.072+0000 7f85027fc700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f84f002ff60 0x7f84f0032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f84ec0096a0 tx=0x7f84ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.077+0000 7f84fb7fe700 1 -- 172.21.15.78:0/1483355223 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84fc0127f0 con 0x7f85040f6510 2024-01-21T08:07:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.280+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85040fc050 con 0x7f85040f6510 2024-01-21T08:07:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.282+0000 7f84fb7fe700 1 -- 172.21.15.78:0/1483355223 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84fc019230 con 0x7f85040f6510 2024-01-21T08:07:26.283 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:26.283 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f84f002ff60 msgr2=0x7f84f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f84f002ff60 0x7f84f0032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f84ec0096a0 tx=0x7f84ec006b40).stop 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 msgr2=0x7f8504106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f8504106730 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f84fc004b70 tx=0x7f84fc004d10).stop 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 shutdown_connections 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f84f002ff60 0x7f84f0032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 --2- 172.21.15.78:0/1483355223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85040f6510 0x7f8504106730 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 >> 172.21.15.78:0/1483355223 conn(0x7f85040f1c70 msgr2=0x7f85040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 shutdown_connections 2024-01-21T08:07:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:26.283+0000 7f85094aa700 1 -- 172.21.15.78:0/1483355223 wait complete. 2024-01-21T08:07:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:24.609161+0000 mgr.smithi046.lmxnwc (mgr.14164) 67 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:26.282094+0000 mon.smithi046 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.78:0/1483355223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:27.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:27.708 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:26.609549+0000 mgr.smithi046.lmxnwc (mgr.14164) 68 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.841+0000 7f1c4f46d700 1 -- 172.21.15.78:0/3782960239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 msgr2=0x7f1c480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.841+0000 7f1c4f46d700 1 --2- 172.21.15.78:0/3782960239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f1c38009fb0 tx=0x7f1c38009b30).stop 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.841+0000 7f1c4f46d700 1 -- 172.21.15.78:0/3782960239 shutdown_connections 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.841+0000 7f1c4f46d700 1 --2- 172.21.15.78:0/3782960239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.841+0000 7f1c4f46d700 1 -- 172.21.15.78:0/3782960239 >> 172.21.15.78:0/3782960239 conn(0x7f1c480f0b70 msgr2=0x7f1c480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.841+0000 7f1c4f46d700 1 -- 172.21.15.78:0/3782960239 shutdown_connections 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.841+0000 7f1c4f46d700 1 -- 172.21.15.78:0/3782960239 wait complete. 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.842+0000 7f1c4f46d700 1 Processor -- start 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.842+0000 7f1c4f46d700 1 -- start start 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.842+0000 7f1c4f46d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.842+0000 7f1c4f46d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480fb820 con 0x7f1c480f5410 2024-01-21T08:07:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.842+0000 7f1c4d209700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.842+0000 7f1c4d209700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:45638/0 (socket says 172.21.15.78:45638) 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.842+0000 7f1c4d209700 1 -- 172.21.15.78:0/4238658001 learned_addr learned my addr 172.21.15.78:0/4238658001 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.843+0000 7f1c4d209700 1 -- 172.21.15.78:0/4238658001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38005040 con 0x7f1c480f5410 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.843+0000 7f1c4d209700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480fd3a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f1c38013510 tx=0x7f1c3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.843+0000 7f1c3dffb700 1 -- 172.21.15.78:0/4238658001 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c38009320 con 0x7f1c480f5410 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.843+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c480fb9c0 con 0x7f1c480f5410 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.843+0000 7f1c3dffb700 1 -- 172.21.15.78:0/4238658001 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c38013f00 con 0x7f1c480f5410 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.843+0000 7f1c3dffb700 1 -- 172.21.15.78:0/4238658001 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c38014d90 con 0x7f1c480f5410 2024-01-21T08:07:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.843+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c480fbe50 con 0x7f1c480f5410 2024-01-21T08:07:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.844+0000 7f1c3dffb700 1 -- 172.21.15.78:0/4238658001 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c380095f0 con 0x7f1c480f5410 2024-01-21T08:07:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.844+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48059100 con 0x7f1c480f5410 2024-01-21T08:07:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.844+0000 7f1c3dffb700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c3402ff60 0x7f1c34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.844+0000 7f1c3dffb700 1 -- 172.21.15.78:0/4238658001 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c38041080 con 0x7f1c480f5410 2024-01-21T08:07:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.844+0000 7f1c4ca08700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c3402ff60 0x7f1c34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.845+0000 7f1c4ca08700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c3402ff60 0x7f1c34032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1c440096a0 tx=0x7f1c44006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:28.849+0000 7f1c3dffb700 1 -- 172.21.15.78:0/4238658001 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c38014420 con 0x7f1c480f5410 2024-01-21T08:07:29.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.055+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c480fae60 con 0x7f1c480f5410 2024-01-21T08:07:29.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.056+0000 7f1c3dffb700 1 -- 172.21.15.78:0/4238658001 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c3801d7f0 con 0x7f1c480f5410 2024-01-21T08:07:29.057 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:29.058 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:29.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c3402ff60 msgr2=0x7f1c34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:29.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c3402ff60 0x7f1c34032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1c440096a0 tx=0x7f1c44006b40).stop 2024-01-21T08:07:29.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 msgr2=0x7f1c480fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480fd3a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f1c38013510 tx=0x7f1c3800b040).stop 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 shutdown_connections 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c3402ff60 0x7f1c34032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 --2- 172.21.15.78:0/4238658001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c480f5410 0x7f1c480fd3a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 >> 172.21.15.78:0/4238658001 conn(0x7f1c480f0b70 msgr2=0x7f1c4805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 shutdown_connections 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:29.057+0000 7f1c4f46d700 1 -- 172.21.15.78:0/4238658001 wait complete. 2024-01-21T08:07:29.060 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:29.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:29.056786+0000 mon.smithi046 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.78:0/4238658001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:30.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:30.404 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:28.609881+0000 mgr.smithi046.lmxnwc (mgr.14164) 69 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.630+0000 7f322ff16700 1 -- 172.21.15.78:0/1554220263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 msgr2=0x7f32280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.630+0000 7f322ff16700 1 --2- 172.21.15.78:0/1554220263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f32280f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3218009960 tx=0x7f3218004d10).stop 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.630+0000 7f322ff16700 1 -- 172.21.15.78:0/1554220263 shutdown_connections 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.630+0000 7f322ff16700 1 --2- 172.21.15.78:0/1554220263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f32280f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.630+0000 7f322ff16700 1 -- 172.21.15.78:0/1554220263 >> 172.21.15.78:0/1554220263 conn(0x7f32280f1c70 msgr2=0x7f32280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.630+0000 7f322ff16700 1 -- 172.21.15.78:0/1554220263 shutdown_connections 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.630+0000 7f322ff16700 1 -- 172.21.15.78:0/1554220263 wait complete. 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322ff16700 1 Processor -- start 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322ff16700 1 -- start start 2024-01-21T08:07:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322ff16700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f3228106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322ff16700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3228106c20 con 0x7f32280f6510 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322dcb2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f3228106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322dcb2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f3228106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:45648/0 (socket says 172.21.15.78:45648) 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322dcb2700 1 -- 172.21.15.78:0/2785129590 learned_addr learned my addr 172.21.15.78:0/2785129590 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322dcb2700 1 -- 172.21.15.78:0/2785129590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3218009640 con 0x7f32280f6510 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.631+0000 7f322dcb2700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f3228106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3218004b70 tx=0x7f3218004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.632+0000 7f32267fc700 1 -- 172.21.15.78:0/2785129590 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3218018070 con 0x7f32280f6510 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.632+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3228106e20 con 0x7f32280f6510 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.632+0000 7f32267fc700 1 -- 172.21.15.78:0/2785129590 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f321800b870 con 0x7f32280f6510 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.632+0000 7f32267fc700 1 -- 172.21.15.78:0/2785129590 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f321801c560 con 0x7f32280f6510 2024-01-21T08:07:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.632+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32281072b0 con 0x7f32280f6510 2024-01-21T08:07:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.633+0000 7f32267fc700 1 -- 172.21.15.78:0/2785129590 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3218013630 con 0x7f32280f6510 2024-01-21T08:07:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.633+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3228045ad0 con 0x7f32280f6510 2024-01-21T08:07:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.633+0000 7f32267fc700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f321002ff60 0x7f3210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.633+0000 7f32267fc700 1 -- 172.21.15.78:0/2785129590 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32180411d0 con 0x7f32280f6510 2024-01-21T08:07:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.633+0000 7f322d4b1700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f321002ff60 0x7f3210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.633+0000 7f322d4b1700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f321002ff60 0x7f3210032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f321c0096a0 tx=0x7f321c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.638+0000 7f32267fc700 1 -- 172.21.15.78:0/2785129590 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3218004810 con 0x7f32280f6510 2024-01-21T08:07:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.844+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32280fbf60 con 0x7f32280f6510 2024-01-21T08:07:31.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.844+0000 7f32267fc700 1 -- 172.21.15.78:0/2785129590 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3218012380 con 0x7f32280f6510 2024-01-21T08:07:31.846 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:31.846 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f321002ff60 msgr2=0x7f3210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f321002ff60 0x7f3210032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f321c0096a0 tx=0x7f321c006b40).stop 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 msgr2=0x7f3228106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f3228106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3218004b70 tx=0x7f3218004d10).stop 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 shutdown_connections 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f321002ff60 0x7f3210032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 --2- 172.21.15.78:0/2785129590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32280f6510 0x7f3228106730 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 >> 172.21.15.78:0/2785129590 conn(0x7f32280f1c70 msgr2=0x7f32280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 shutdown_connections 2024-01-21T08:07:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:31.846+0000 7f322ff16700 1 -- 172.21.15.78:0/2785129590 wait complete. 2024-01-21T08:07:31.849 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:30.610211+0000 mgr.smithi046.lmxnwc (mgr.14164) 70 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:31.845373+0000 mon.smithi046 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.78:0/2785129590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:33.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:33.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:34.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.461+0000 7f909cf2f700 1 -- 172.21.15.78:0/2001348610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 msgr2=0x7f90980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.461+0000 7f909cf2f700 1 --2- 172.21.15.78:0/2001348610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f90980f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f9088009fb0 tx=0x7f9088009b30).stop 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.461+0000 7f909cf2f700 1 -- 172.21.15.78:0/2001348610 shutdown_connections 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.461+0000 7f909cf2f700 1 --2- 172.21.15.78:0/2001348610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f90980f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.461+0000 7f909cf2f700 1 -- 172.21.15.78:0/2001348610 >> 172.21.15.78:0/2001348610 conn(0x7f90980f0b70 msgr2=0x7f90980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.461+0000 7f909cf2f700 1 -- 172.21.15.78:0/2001348610 shutdown_connections 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.461+0000 7f909cf2f700 1 -- 172.21.15.78:0/2001348610 wait complete. 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909cf2f700 1 Processor -- start 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909cf2f700 1 -- start start 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909cf2f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f9098108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909cf2f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098108e10 con 0x7f90980f5410 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f9098108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:34.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f9098108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:42982/0 (socket says 172.21.15.78:42982) 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909659c700 1 -- 172.21.15.78:0/36111947 learned_addr learned my addr 172.21.15.78:0/36111947 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909659c700 1 -- 172.21.15.78:0/36111947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088005040 con 0x7f90980f5410 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.462+0000 7f909659c700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f9098108920 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9088013510 tx=0x7f908800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.463+0000 7f9086ffd700 1 -- 172.21.15.78:0/36111947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9088009320 con 0x7f90980f5410 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.463+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9098109010 con 0x7f90980f5410 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.463+0000 7f9086ffd700 1 -- 172.21.15.78:0/36111947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9088013f00 con 0x7f90980f5410 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.463+0000 7f9086ffd700 1 -- 172.21.15.78:0/36111947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9088014d90 con 0x7f90980f5410 2024-01-21T08:07:34.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.463+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90981094a0 con 0x7f90980f5410 2024-01-21T08:07:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.463+0000 7f9086ffd700 1 -- 172.21.15.78:0/36111947 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90880095f0 con 0x7f90980f5410 2024-01-21T08:07:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.464+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9098045ad0 con 0x7f90980f5410 2024-01-21T08:07:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.464+0000 7f9086ffd700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f908002ff60 0x7f9080032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.464+0000 7f9086ffd700 1 -- 172.21.15.78:0/36111947 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90880410c0 con 0x7f90980f5410 2024-01-21T08:07:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.464+0000 7f9095d9b700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f908002ff60 0x7f9080032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.464+0000 7f9095d9b700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f908002ff60 0x7f9080032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f908c0096a0 tx=0x7f908c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:34.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.469+0000 7f9086ffd700 1 -- 172.21.15.78:0/36111947 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9088014420 con 0x7f90980f5410 2024-01-21T08:07:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.670+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90980f9d80 con 0x7f90980f5410 2024-01-21T08:07:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.671+0000 7f9086ffd700 1 -- 172.21.15.78:0/36111947 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f908801d7f0 con 0x7f90980f5410 2024-01-21T08:07:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:34.673 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f908002ff60 msgr2=0x7f9080032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f908002ff60 0x7f9080032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f908c0096a0 tx=0x7f908c006b40).stop 2024-01-21T08:07:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 msgr2=0x7f9098108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f9098108920 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9088013510 tx=0x7f908800b040).stop 2024-01-21T08:07:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 shutdown_connections 2024-01-21T08:07:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f908002ff60 0x7f9080032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 --2- 172.21.15.78:0/36111947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90980f5410 0x7f9098108920 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 >> 172.21.15.78:0/36111947 conn(0x7f90980f0b70 msgr2=0x7f90980f2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 shutdown_connections 2024-01-21T08:07:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:34.672+0000 7f909cf2f700 1 -- 172.21.15.78:0/36111947 wait complete. 2024-01-21T08:07:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:34.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:32.610636+0000 mgr.smithi046.lmxnwc (mgr.14164) 71 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:34.671853+0000 mon.smithi046 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.78:0/36111947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:36.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:36.114 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:36.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:34.610921+0000 mgr.smithi046.lmxnwc (mgr.14164) 72 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:37.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.350+0000 7fc67759e700 1 -- 172.21.15.78:0/3802080540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 msgr2=0x7fc6780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:37.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.350+0000 7fc67759e700 1 --2- 172.21.15.78:0/3802080540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6780f55c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fc668009fb0 tx=0x7fc668009b30).stop 2024-01-21T08:07:37.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.350+0000 7fc67759e700 1 -- 172.21.15.78:0/3802080540 shutdown_connections 2024-01-21T08:07:37.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.350+0000 7fc67759e700 1 --2- 172.21.15.78:0/3802080540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6780f55c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:37.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.350+0000 7fc67759e700 1 -- 172.21.15.78:0/3802080540 >> 172.21.15.78:0/3802080540 conn(0x7fc6780f09a0 msgr2=0x7fc6780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67759e700 1 -- 172.21.15.78:0/3802080540 shutdown_connections 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67759e700 1 -- 172.21.15.78:0/3802080540 wait complete. 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67759e700 1 Processor -- start 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67759e700 1 -- start start 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6781091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67759e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678060220 con 0x7fc6780f51a0 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6781091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6781091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:43010/0 (socket says 172.21.15.78:43010) 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.351+0000 7fc67659c700 1 -- 172.21.15.78:0/2419156175 learned_addr learned my addr 172.21.15.78:0/2419156175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.352+0000 7fc67659c700 1 -- 172.21.15.78:0/2419156175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668005040 con 0x7fc6780f51a0 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.352+0000 7fc67659c700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6781091d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc668013510 tx=0x7fc66800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.352+0000 7fc666ffd700 1 -- 172.21.15.78:0/2419156175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc668009320 con 0x7fc6780f51a0 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.352+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc678060420 con 0x7fc6780f51a0 2024-01-21T08:07:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.352+0000 7fc666ffd700 1 -- 172.21.15.78:0/2419156175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc668013f00 con 0x7fc6780f51a0 2024-01-21T08:07:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.352+0000 7fc666ffd700 1 -- 172.21.15.78:0/2419156175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc668014dc0 con 0x7fc6780f51a0 2024-01-21T08:07:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.352+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc678109880 con 0x7fc6780f51a0 2024-01-21T08:07:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.353+0000 7fc666ffd700 1 -- 172.21.15.78:0/2419156175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6680095f0 con 0x7fc6780f51a0 2024-01-21T08:07:37.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.353+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc678059100 con 0x7fc6780f51a0 2024-01-21T08:07:37.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.353+0000 7fc666ffd700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc66002ff60 0x7fc660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:37.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.353+0000 7fc666ffd700 1 -- 172.21.15.78:0/2419156175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6680410e0 con 0x7fc6780f51a0 2024-01-21T08:07:37.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.353+0000 7fc675d9b700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc66002ff60 0x7fc660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:37.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.354+0000 7fc675d9b700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc66002ff60 0x7fc660032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc66c0096a0 tx=0x7fc66c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:37.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.358+0000 7fc666ffd700 1 -- 172.21.15.78:0/2419156175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc668014420 con 0x7fc6780f51a0 2024-01-21T08:07:37.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.564+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6780fabe0 con 0x7fc6780f51a0 2024-01-21T08:07:37.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.564+0000 7fc666ffd700 1 -- 172.21.15.78:0/2419156175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc66801c920 con 0x7fc6780f51a0 2024-01-21T08:07:37.566 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:37.566 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc66002ff60 msgr2=0x7fc660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc66002ff60 0x7fc660032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc66c0096a0 tx=0x7fc66c006b40).stop 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 msgr2=0x7fc6781091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6781091d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc668013510 tx=0x7fc66800b040).stop 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 shutdown_connections 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc66002ff60 0x7fc660032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 --2- 172.21.15.78:0/2419156175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6780f51a0 0x7fc6781091d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 >> 172.21.15.78:0/2419156175 conn(0x7fc6780f09a0 msgr2=0x7fc6780f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 shutdown_connections 2024-01-21T08:07:37.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:37.566+0000 7fc67759e700 1 -- 172.21.15.78:0/2419156175 wait complete. 2024-01-21T08:07:37.569 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:36.611257+0000 mgr.smithi046.lmxnwc (mgr.14164) 73 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:37.565472+0000 mon.smithi046 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.78:0/2419156175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:38.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:38.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:40.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.190+0000 7f75489b8700 1 -- 172.21.15.78:0/3351664409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 msgr2=0x7f75440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:40.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.190+0000 7f75489b8700 1 --2- 172.21.15.78:0/3351664409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f75440f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f752c009fb0 tx=0x7f752c009b30).stop 2024-01-21T08:07:40.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.190+0000 7f75489b8700 1 -- 172.21.15.78:0/3351664409 shutdown_connections 2024-01-21T08:07:40.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.190+0000 7f75489b8700 1 --2- 172.21.15.78:0/3351664409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f75440f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:40.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.190+0000 7f75489b8700 1 -- 172.21.15.78:0/3351664409 >> 172.21.15.78:0/3351664409 conn(0x7f75440f1c70 msgr2=0x7f75440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:40.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.190+0000 7f75489b8700 1 -- 172.21.15.78:0/3351664409 shutdown_connections 2024-01-21T08:07:40.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.190+0000 7f75489b8700 1 -- 172.21.15.78:0/3351664409 wait complete. 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f75489b8700 1 Processor -- start 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f75489b8700 1 -- start start 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f75489b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f7544106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f75489b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544106c20 con 0x7f75440f6510 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f754259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f7544106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f754259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f7544106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:43022/0 (socket says 172.21.15.78:43022) 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f754259c700 1 -- 172.21.15.78:0/2401828802 learned_addr learned my addr 172.21.15.78:0/2401828802 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f754259c700 1 -- 172.21.15.78:0/2401828802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c005040 con 0x7f75440f6510 2024-01-21T08:07:40.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.191+0000 7f754259c700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f7544106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f752c013510 tx=0x7f752c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:40.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.192+0000 7f753affd700 1 -- 172.21.15.78:0/2401828802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c009320 con 0x7f75440f6510 2024-01-21T08:07:40.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.192+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7544106e20 con 0x7f75440f6510 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.192+0000 7f753affd700 1 -- 172.21.15.78:0/2401828802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f752c013f00 con 0x7f75440f6510 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.192+0000 7f753affd700 1 -- 172.21.15.78:0/2401828802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c014d90 con 0x7f75440f6510 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.192+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75441072b0 con 0x7f75440f6510 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.192+0000 7f753affd700 1 -- 172.21.15.78:0/2401828802 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f752c0095f0 con 0x7f75440f6510 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.192+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7544045ad0 con 0x7f75440f6510 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.193+0000 7f753affd700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f753002ff60 0x7f7530032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.193+0000 7f753affd700 1 -- 172.21.15.78:0/2401828802 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f752c040fc0 con 0x7f75440f6510 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.193+0000 7f7541d9b700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f753002ff60 0x7f7530032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:40.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.193+0000 7f7541d9b700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f753002ff60 0x7f7530032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f75340096a0 tx=0x7f7534006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:40.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.198+0000 7f753affd700 1 -- 172.21.15.78:0/2401828802 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f752c014420 con 0x7f75440f6510 2024-01-21T08:07:40.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.405+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75440fbf60 con 0x7f75440f6510 2024-01-21T08:07:40.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.405+0000 7f753affd700 1 -- 172.21.15.78:0/2401828802 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f752c010230 con 0x7f75440f6510 2024-01-21T08:07:40.407 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:40.407 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f753002ff60 msgr2=0x7f7530032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f753002ff60 0x7f7530032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f75340096a0 tx=0x7f7534006b40).stop 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 msgr2=0x7f7544106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f7544106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f752c013510 tx=0x7f752c00b040).stop 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 shutdown_connections 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f753002ff60 0x7f7530032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 --2- 172.21.15.78:0/2401828802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75440f6510 0x7f7544106730 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 >> 172.21.15.78:0/2401828802 conn(0x7f75440f1c70 msgr2=0x7f75440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:40.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 shutdown_connections 2024-01-21T08:07:40.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:40.407+0000 7f75489b8700 1 -- 172.21.15.78:0/2401828802 wait complete. 2024-01-21T08:07:40.410 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:38.611666+0000 mgr.smithi046.lmxnwc (mgr.14164) 74 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:40.406447+0000 mon.smithi046 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.78:0/2401828802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:41.834 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:41.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:40.611889+0000 mgr.smithi046.lmxnwc (mgr.14164) 75 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:43.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 -- 172.21.15.78:0/2693348395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 msgr2=0x7f6b400fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 --2- 172.21.15.78:0/2693348395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b400fb380 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6b30009fb0 tx=0x7f6b30009b30).stop 2024-01-21T08:07:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 -- 172.21.15.78:0/2693348395 shutdown_connections 2024-01-21T08:07:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 --2- 172.21.15.78:0/2693348395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b400fb380 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 -- 172.21.15.78:0/2693348395 >> 172.21.15.78:0/2693348395 conn(0x7f6b400f09a0 msgr2=0x7f6b400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:43.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 -- 172.21.15.78:0/2693348395 shutdown_connections 2024-01-21T08:07:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 -- 172.21.15.78:0/2693348395 wait complete. 2024-01-21T08:07:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 Processor -- start 2024-01-21T08:07:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 -- start start 2024-01-21T08:07:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b401086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.012+0000 7f6b4489f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b40108bc0 con 0x7f6b40060180 2024-01-21T08:07:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b3f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b401086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:43.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b3f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b401086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:40250/0 (socket says 172.21.15.78:40250) 2024-01-21T08:07:43.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b3f7fe700 1 -- 172.21.15.78:0/2644405785 learned_addr learned my addr 172.21.15.78:0/2644405785 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:43.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b3f7fe700 1 -- 172.21.15.78:0/2644405785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30005040 con 0x7f6b40060180 2024-01-21T08:07:43.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b3f7fe700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b401086d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6b30013510 tx=0x7f6b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:43.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b27fff700 1 -- 172.21.15.78:0/2644405785 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b30009320 con 0x7f6b40060180 2024-01-21T08:07:43.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b40108dc0 con 0x7f6b40060180 2024-01-21T08:07:43.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b27fff700 1 -- 172.21.15.78:0/2644405785 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b30013f00 con 0x7f6b40060180 2024-01-21T08:07:43.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b27fff700 1 -- 172.21.15.78:0/2644405785 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b30014d90 con 0x7f6b40060180 2024-01-21T08:07:43.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.013+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b40109250 con 0x7f6b40060180 2024-01-21T08:07:43.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.014+0000 7f6b27fff700 1 -- 172.21.15.78:0/2644405785 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b30014420 con 0x7f6b40060180 2024-01-21T08:07:43.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.014+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b40045ad0 con 0x7f6b40060180 2024-01-21T08:07:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.014+0000 7f6b27fff700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6b2802ff60 0x7f6b28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.015+0000 7f6b27fff700 1 -- 172.21.15.78:0/2644405785 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b30040fb0 con 0x7f6b40060180 2024-01-21T08:07:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.015+0000 7f6b3effd700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6b2802ff60 0x7f6b28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.015+0000 7f6b3effd700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6b2802ff60 0x7f6b28032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6b340096a0 tx=0x7f6b34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:43.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.019+0000 7f6b27fff700 1 -- 172.21.15.78:0/2644405785 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b300095f0 con 0x7f6b40060180 2024-01-21T08:07:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.225+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b4010a100 con 0x7f6b40060180 2024-01-21T08:07:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.227+0000 7f6b27fff700 1 -- 172.21.15.78:0/2644405785 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b3001db70 con 0x7f6b40060180 2024-01-21T08:07:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.227+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6b2802ff60 msgr2=0x7f6b28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.227+0000 7f6b4489f700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6b2802ff60 0x7f6b28032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6b340096a0 tx=0x7f6b34006b40).stop 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.227+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 msgr2=0x7f6b401086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.227+0000 7f6b4489f700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b401086d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6b30013510 tx=0x7f6b3000b040).stop 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.228+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 shutdown_connections 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.228+0000 7f6b4489f700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6b2802ff60 0x7f6b28032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.228+0000 7f6b4489f700 1 --2- 172.21.15.78:0/2644405785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b40060180 0x7f6b401086d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.228+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 >> 172.21.15.78:0/2644405785 conn(0x7f6b400f09a0 msgr2=0x7f6b400f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.228+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 shutdown_connections 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:43.228+0000 7f6b4489f700 1 -- 172.21.15.78:0/2644405785 wait complete. 2024-01-21T08:07:43.231 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:43.227939+0000 mon.smithi046 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.78:0/2644405785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:44.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:44.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:44.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:42.612170+0000 mgr.smithi046.lmxnwc (mgr.14164) 76 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:44.612534+0000 mgr.smithi046.lmxnwc (mgr.14164) 77 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:45.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.906+0000 7f573b437700 1 -- 172.21.15.78:0/2710796396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 msgr2=0x7f57340f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:45.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.906+0000 7f573b437700 1 --2- 172.21.15.78:0/2710796396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f57340f4be0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5724009fb0 tx=0x7f5724009b30).stop 2024-01-21T08:07:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.906+0000 7f573b437700 1 -- 172.21.15.78:0/2710796396 shutdown_connections 2024-01-21T08:07:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.906+0000 7f573b437700 1 --2- 172.21.15.78:0/2710796396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f57340f4be0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.906+0000 7f573b437700 1 -- 172.21.15.78:0/2710796396 >> 172.21.15.78:0/2710796396 conn(0x7f57340f02e0 msgr2=0x7f57340f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.906+0000 7f573b437700 1 -- 172.21.15.78:0/2710796396 shutdown_connections 2024-01-21T08:07:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.906+0000 7f573b437700 1 -- 172.21.15.78:0/2710796396 wait complete. 2024-01-21T08:07:45.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.907+0000 7f573b437700 1 Processor -- start 2024-01-21T08:07:45.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.907+0000 7f573b437700 1 -- start start 2024-01-21T08:07:45.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.907+0000 7f573b437700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f5734106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.907+0000 7f573b437700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734106bf0 con 0x7f57340f47c0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.907+0000 7f57391d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f5734106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f57391d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f5734106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:40270/0 (socket says 172.21.15.78:40270) 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f57391d3700 1 -- 172.21.15.78:0/1869674539 learned_addr learned my addr 172.21.15.78:0/1869674539 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f57391d3700 1 -- 172.21.15.78:0/1869674539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5724005040 con 0x7f57340f47c0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f57391d3700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f5734106700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5724013510 tx=0x7f572400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f5729ffb700 1 -- 172.21.15.78:0/1869674539 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5724009320 con 0x7f57340f47c0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5734106df0 con 0x7f57340f47c0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f5729ffb700 1 -- 172.21.15.78:0/1869674539 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5724013f00 con 0x7f57340f47c0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f5729ffb700 1 -- 172.21.15.78:0/1869674539 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5724014d90 con 0x7f57340f47c0 2024-01-21T08:07:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.908+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5734107280 con 0x7f57340f47c0 2024-01-21T08:07:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.909+0000 7f5729ffb700 1 -- 172.21.15.78:0/1869674539 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57240263a0 con 0x7f57340f47c0 2024-01-21T08:07:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.909+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5734045ad0 con 0x7f57340f47c0 2024-01-21T08:07:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.909+0000 7f5729ffb700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f572002ff60 0x7f5720032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.909+0000 7f5729ffb700 1 -- 172.21.15.78:0/1869674539 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57240412d0 con 0x7f57340f47c0 2024-01-21T08:07:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.909+0000 7f57389d2700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f572002ff60 0x7f5720032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.910+0000 7f57389d2700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f572002ff60 0x7f5720032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f57300096a0 tx=0x7f5730006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:45.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:45.915+0000 7f5729ffb700 1 -- 172.21.15.78:0/1869674539 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f572401cb10 con 0x7f57340f47c0 2024-01-21T08:07:46.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.120+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5734108130 con 0x7f57340f47c0 2024-01-21T08:07:46.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.122+0000 7f5729ffb700 1 -- 172.21.15.78:0/1869674539 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f572401cb10 con 0x7f57340f47c0 2024-01-21T08:07:46.124 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:46.124 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f572002ff60 msgr2=0x7f5720032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f572002ff60 0x7f5720032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f57300096a0 tx=0x7f5730006b40).stop 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 msgr2=0x7f5734106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f5734106700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5724013510 tx=0x7f572400b040).stop 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 shutdown_connections 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f572002ff60 0x7f5720032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 --2- 172.21.15.78:0/1869674539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57340f47c0 0x7f5734106700 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 >> 172.21.15.78:0/1869674539 conn(0x7f57340f02e0 msgr2=0x7f57340f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 shutdown_connections 2024-01-21T08:07:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:46.123+0000 7f573b437700 1 -- 172.21.15.78:0/1869674539 wait complete. 2024-01-21T08:07:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:46.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:46.122937+0000 mon.smithi046 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.78:0/1869674539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:47.536 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:47.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:46.612791+0000 mgr.smithi046.lmxnwc (mgr.14164) 78 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 -- 172.21.15.78:0/2842266903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 msgr2=0x7f4b8c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 --2- 172.21.15.78:0/2842266903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f4be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4b7c009fb0 tx=0x7f4b7c009b30).stop 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 -- 172.21.15.78:0/2842266903 shutdown_connections 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 --2- 172.21.15.78:0/2842266903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f4be0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 -- 172.21.15.78:0/2842266903 >> 172.21.15.78:0/2842266903 conn(0x7f4b8c0f02e0 msgr2=0x7f4b8c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 -- 172.21.15.78:0/2842266903 shutdown_connections 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 -- 172.21.15.78:0/2842266903 wait complete. 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 Processor -- start 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 -- start start 2024-01-21T08:07:48.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:48.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.815+0000 7f4b9158a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0f57f0 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b8affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:48.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b8affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:40278/0 (socket says 172.21.15.78:40278) 2024-01-21T08:07:48.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b8affd700 1 -- 172.21.15.78:0/1013531365 learned_addr learned my addr 172.21.15.78:0/1013531365 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:48.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b8affd700 1 -- 172.21.15.78:0/1013531365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c005040 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b8affd700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f7370 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f4b7c014000 tx=0x7f4b7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b737fe700 1 -- 172.21.15.78:0/1013531365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b7c0095f0 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c0f5990 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b737fe700 1 -- 172.21.15.78:0/1013531365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b7c007e70 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.816+0000 7f4b737fe700 1 -- 172.21.15.78:0/1013531365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b7c014ea0 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.817+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c0f5e20 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.817+0000 7f4b737fe700 1 -- 172.21.15.78:0/1013531365 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b7c025070 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.817+0000 7f4b737fe700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4b7402ff60 0x7f4b74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.817+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c059100 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.817+0000 7f4b737fe700 1 -- 172.21.15.78:0/1013531365 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b7c040970 con 0x7f4b8c0f47c0 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.818+0000 7f4b8a7fc700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4b7402ff60 0x7f4b74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:48.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.818+0000 7f4b8a7fc700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4b7402ff60 0x7f4b74032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4b800096a0 tx=0x7f4b80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:48.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:48.822+0000 7f4b737fe700 1 -- 172.21.15.78:0/1013531365 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b7c0079c0 con 0x7f4b8c0f47c0 2024-01-21T08:07:49.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.024+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b8c0fbf60 con 0x7f4b8c0f47c0 2024-01-21T08:07:49.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.024+0000 7f4b737fe700 1 -- 172.21.15.78:0/1013531365 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b7c00f300 con 0x7f4b8c0f47c0 2024-01-21T08:07:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:49.026 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4b7402ff60 msgr2=0x7f4b74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4b7402ff60 0x7f4b74032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4b800096a0 tx=0x7f4b80006b40).stop 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 msgr2=0x7f4b8c0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f7370 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f4b7c014000 tx=0x7f4b7c00b040).stop 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 shutdown_connections 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4b7402ff60 0x7f4b74032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 --2- 172.21.15.78:0/1013531365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8c0f47c0 0x7f4b8c0f7370 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 >> 172.21.15.78:0/1013531365 conn(0x7f4b8c0f02e0 msgr2=0x7f4b8c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 shutdown_connections 2024-01-21T08:07:49.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:49.026+0000 7f4b9158a700 1 -- 172.21.15.78:0/1013531365 wait complete. 2024-01-21T08:07:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:49.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:48.613140+0000 mgr.smithi046.lmxnwc (mgr.14164) 79 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:49.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:49.025515+0000 mon.smithi046 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.78:0/1013531365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:50.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:50.358 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.591+0000 7fbc298a7700 1 -- 172.21.15.78:0/1296389051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 msgr2=0x7fbc240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.591+0000 7fbc298a7700 1 --2- 172.21.15.78:0/1296389051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc240f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fbc0c009fb0 tx=0x7fbc0c009b30).stop 2024-01-21T08:07:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.591+0000 7fbc298a7700 1 -- 172.21.15.78:0/1296389051 shutdown_connections 2024-01-21T08:07:51.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.591+0000 7fbc298a7700 1 --2- 172.21.15.78:0/1296389051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc240f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:51.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.591+0000 7fbc298a7700 1 -- 172.21.15.78:0/1296389051 >> 172.21.15.78:0/1296389051 conn(0x7fbc240f0b70 msgr2=0x7fbc240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:51.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.591+0000 7fbc298a7700 1 -- 172.21.15.78:0/1296389051 shutdown_connections 2024-01-21T08:07:51.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.591+0000 7fbc298a7700 1 -- 172.21.15.78:0/1296389051 wait complete. 2024-01-21T08:07:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc298a7700 1 Processor -- start 2024-01-21T08:07:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc298a7700 1 -- start start 2024-01-21T08:07:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc298a7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc2410b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc298a7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc24107210 con 0x7fbc240f5410 2024-01-21T08:07:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc22ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc2410b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:51.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc22ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc2410b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:40300/0 (socket says 172.21.15.78:40300) 2024-01-21T08:07:51.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc22ffd700 1 -- 172.21.15.78:0/2179513669 learned_addr learned my addr 172.21.15.78:0/2179513669 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:51.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.592+0000 7fbc22ffd700 1 -- 172.21.15.78:0/2179513669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c005040 con 0x7fbc240f5410 2024-01-21T08:07:51.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.593+0000 7fbc22ffd700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc2410b820 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbc0c013510 tx=0x7fbc0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:51.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.593+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/2179513669 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc0c009320 con 0x7fbc240f5410 2024-01-21T08:07:51.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.593+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc24060220 con 0x7fbc240f5410 2024-01-21T08:07:51.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.593+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/2179513669 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc0c013f00 con 0x7fbc240f5410 2024-01-21T08:07:51.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.593+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/2179513669 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc0c014d90 con 0x7fbc240f5410 2024-01-21T08:07:51.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.593+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2410bd10 con 0x7fbc240f5410 2024-01-21T08:07:51.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.594+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/2179513669 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc0c0263a0 con 0x7fbc240f5410 2024-01-21T08:07:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.594+0000 7fbc1b7fe700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbc1002ffb0 0x7fbc10032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.594+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/2179513669 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc0c040fb0 con 0x7fbc240f5410 2024-01-21T08:07:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.594+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc24059100 con 0x7fbc240f5410 2024-01-21T08:07:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.594+0000 7fbc227fc700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbc1002ffb0 0x7fbc10032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.594+0000 7fbc227fc700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbc1002ffb0 0x7fbc10032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbc140096a0 tx=0x7fbc14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:51.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.599+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/2179513669 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc0c012920 con 0x7fbc240f5410 2024-01-21T08:07:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.801+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc24045ad0 con 0x7fbc240f5410 2024-01-21T08:07:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.802+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/2179513669 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc0c02b020 con 0x7fbc240f5410 2024-01-21T08:07:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:51.804 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbc1002ffb0 msgr2=0x7fbc10032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbc1002ffb0 0x7fbc10032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbc140096a0 tx=0x7fbc14006b40).stop 2024-01-21T08:07:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 msgr2=0x7fbc2410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc2410b820 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbc0c013510 tx=0x7fbc0c00b040).stop 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 shutdown_connections 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbc1002ffb0 0x7fbc10032470 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 --2- 172.21.15.78:0/2179513669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc240f5410 0x7fbc2410b820 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 >> 172.21.15.78:0/2179513669 conn(0x7fbc240f0b70 msgr2=0x7fbc240f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 shutdown_connections 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:51.803+0000 7fbc298a7700 1 -- 172.21.15.78:0/2179513669 wait complete. 2024-01-21T08:07:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:51.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:50.613465+0000 mgr.smithi046.lmxnwc (mgr.14164) 80 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:52.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:51.802874+0000 mon.smithi046 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.78:0/2179513669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:53.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:53.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:53.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:52.613774+0000 mgr.smithi046.lmxnwc (mgr.14164) 81 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.396+0000 7f86c2fa2700 1 -- 172.21.15.78:0/1473783041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 msgr2=0x7f86bc0f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.396+0000 7f86c2fa2700 1 --2- 172.21.15.78:0/1473783041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc0f5c30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f86b8009fb0 tx=0x7f86b8009b30).stop 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.396+0000 7f86c2fa2700 1 -- 172.21.15.78:0/1473783041 shutdown_connections 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.396+0000 7f86c2fa2700 1 --2- 172.21.15.78:0/1473783041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc0f5c30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.396+0000 7f86c2fa2700 1 -- 172.21.15.78:0/1473783041 >> 172.21.15.78:0/1473783041 conn(0x7f86bc0f0120 msgr2=0x7f86bc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c2fa2700 1 -- 172.21.15.78:0/1473783041 shutdown_connections 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c2fa2700 1 -- 172.21.15.78:0/1473783041 wait complete. 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c2fa2700 1 Processor -- start 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c2fa2700 1 -- start start 2024-01-21T08:07:54.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c2fa2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c2fa2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc0f51d0 con 0x7f86bc0f5810 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c1fa0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c1fa0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:45288/0 (socket says 172.21.15.78:45288) 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.397+0000 7f86c1fa0700 1 -- 172.21.15.78:0/2065678176 learned_addr learned my addr 172.21.15.78:0/2065678176 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.398+0000 7f86c1fa0700 1 -- 172.21.15.78:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8005040 con 0x7f86bc0f5810 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.398+0000 7f86c1fa0700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc109200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f86b8004dd0 tx=0x7f86b8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.398+0000 7f86ae7fc700 1 -- 172.21.15.78:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86b8018070 con 0x7f86bc0f5810 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.398+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86bc0f53d0 con 0x7f86bc0f5810 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.398+0000 7f86ae7fc700 1 -- 172.21.15.78:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86b8013410 con 0x7f86bc0f5810 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.398+0000 7f86ae7fc700 1 -- 172.21.15.78:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86b801c3c0 con 0x7f86bc0f5810 2024-01-21T08:07:54.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.398+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86bc1098b0 con 0x7f86bc0f5810 2024-01-21T08:07:54.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.399+0000 7f86ae7fc700 1 -- 172.21.15.78:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86b801d420 con 0x7f86bc0f5810 2024-01-21T08:07:54.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.399+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86bc059100 con 0x7f86bc0f5810 2024-01-21T08:07:54.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.399+0000 7f86ae7fc700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f86a802ff60 0x7f86a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:54.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.399+0000 7f86ae7fc700 1 -- 172.21.15.78:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86b80413b0 con 0x7f86bc0f5810 2024-01-21T08:07:54.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.399+0000 7f86c179f700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f86a802ff60 0x7f86a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:54.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.400+0000 7f86c179f700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f86a802ff60 0x7f86a8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f86b00096a0 tx=0x7f86b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:54.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.404+0000 7f86ae7fc700 1 -- 172.21.15.78:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86b801d820 con 0x7f86bc0f5810 2024-01-21T08:07:54.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.607+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86bc045ad0 con 0x7f86bc0f5810 2024-01-21T08:07:54.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.608+0000 7f86ae7fc700 1 -- 172.21.15.78:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f86b801b370 con 0x7f86bc0f5810 2024-01-21T08:07:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:54.610 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f86a802ff60 msgr2=0x7f86a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f86a802ff60 0x7f86a8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f86b00096a0 tx=0x7f86b0006b40).stop 2024-01-21T08:07:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 msgr2=0x7f86bc109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc109200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f86b8004dd0 tx=0x7f86b8009b30).stop 2024-01-21T08:07:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 shutdown_connections 2024-01-21T08:07:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f86a802ff60 0x7f86a8032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 --2- 172.21.15.78:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86bc0f5810 0x7f86bc109200 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 >> 172.21.15.78:0/2065678176 conn(0x7f86bc0f0120 msgr2=0x7f86bc0f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 shutdown_connections 2024-01-21T08:07:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:54.609+0000 7f86c2fa2700 1 -- 172.21.15.78:0/2065678176 wait complete. 2024-01-21T08:07:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:54.608848+0000 mon.smithi046 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.78:0/2065678176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:56.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:56.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:07:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:54.614010+0000 mgr.smithi046.lmxnwc (mgr.14164) 82 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.247+0000 7fb721120700 1 -- 172.21.15.78:0/3357043360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 msgr2=0x7fb71c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.247+0000 7fb721120700 1 --2- 172.21.15.78:0/3357043360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c0f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fb70c009fb0 tx=0x7fb70c009b30).stop 2024-01-21T08:07:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.247+0000 7fb721120700 1 -- 172.21.15.78:0/3357043360 shutdown_connections 2024-01-21T08:07:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.247+0000 7fb721120700 1 --2- 172.21.15.78:0/3357043360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c0f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.247+0000 7fb721120700 1 -- 172.21.15.78:0/3357043360 >> 172.21.15.78:0/3357043360 conn(0x7fb71c0f0b70 msgr2=0x7fb71c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.247+0000 7fb721120700 1 -- 172.21.15.78:0/3357043360 shutdown_connections 2024-01-21T08:07:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.247+0000 7fb721120700 1 -- 172.21.15.78:0/3357043360 wait complete. 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb721120700 1 Processor -- start 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb721120700 1 -- start start 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb721120700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c10b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb721120700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c1071e0 con 0x7fb71c0f5410 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb71ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c10b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb71ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c10b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:45304/0 (socket says 172.21.15.78:45304) 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb71ad9d700 1 -- 172.21.15.78:0/2934020299 learned_addr learned my addr 172.21.15.78:0/2934020299 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.248+0000 7fb71ad9d700 1 -- 172.21.15.78:0/2934020299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c005040 con 0x7fb71c0f5410 2024-01-21T08:07:57.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.249+0000 7fb71ad9d700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c10b7f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb70c013510 tx=0x7fb70c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:57.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.249+0000 7fb7137fe700 1 -- 172.21.15.78:0/2934020299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb70c009320 con 0x7fb71c0f5410 2024-01-21T08:07:57.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.249+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c060220 con 0x7fb71c0f5410 2024-01-21T08:07:57.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.249+0000 7fb7137fe700 1 -- 172.21.15.78:0/2934020299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb70c013f00 con 0x7fb71c0f5410 2024-01-21T08:07:57.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.249+0000 7fb7137fe700 1 -- 172.21.15.78:0/2934020299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb70c014d90 con 0x7fb71c0f5410 2024-01-21T08:07:57.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.249+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c10bce0 con 0x7fb71c0f5410 2024-01-21T08:07:57.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.250+0000 7fb7137fe700 1 -- 172.21.15.78:0/2934020299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb70c0095f0 con 0x7fb71c0f5410 2024-01-21T08:07:57.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.250+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c045ad0 con 0x7fb71c0f5410 2024-01-21T08:07:57.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.250+0000 7fb7137fe700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb70802ff60 0x7fb708032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:07:57.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.250+0000 7fb7137fe700 1 -- 172.21.15.78:0/2934020299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb70c040fe0 con 0x7fb71c0f5410 2024-01-21T08:07:57.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.250+0000 7fb71a59c700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb70802ff60 0x7fb708032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:07:57.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.251+0000 7fb71a59c700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb70802ff60 0x7fb708032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb7040096a0 tx=0x7fb704006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:07:57.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.255+0000 7fb7137fe700 1 -- 172.21.15.78:0/2934020299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb70c01d7f0 con 0x7fb71c0f5410 2024-01-21T08:07:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.460+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb71c10c4a0 con 0x7fb71c0f5410 2024-01-21T08:07:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.461+0000 7fb7137fe700 1 -- 172.21.15.78:0/2934020299 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb70c010230 con 0x7fb71c0f5410 2024-01-21T08:07:57.463 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:07:57.463 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:07:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.462+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb70802ff60 msgr2=0x7fb708032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.462+0000 7fb721120700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb70802ff60 0x7fb708032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb7040096a0 tx=0x7fb704006b40).stop 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 msgr2=0x7fb71c10b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c10b7f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb70c013510 tx=0x7fb70c00b040).stop 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 shutdown_connections 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb70802ff60 0x7fb708032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 --2- 172.21.15.78:0/2934020299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb71c0f5410 0x7fb71c10b7f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 >> 172.21.15.78:0/2934020299 conn(0x7fb71c0f0b70 msgr2=0x7fb71c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 shutdown_connections 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:07:57.463+0000 7fb721120700 1 -- 172.21.15.78:0/2934020299 wait complete. 2024-01-21T08:07:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:07:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:56.614226+0000 mgr.smithi046.lmxnwc (mgr.14164) 83 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:07:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:07:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:07:57.461996+0000 mon.smithi046 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.78:0/2934020299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:07:58.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:07:58.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:00.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.135+0000 7fcbe68ee700 1 -- 172.21.15.78:0/3766453672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 msgr2=0x7fcbe00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:00.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.135+0000 7fcbe68ee700 1 --2- 172.21.15.78:0/3766453672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe00f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fcbd0009fb0 tx=0x7fcbd0009b30).stop 2024-01-21T08:08:00.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.135+0000 7fcbe68ee700 1 -- 172.21.15.78:0/3766453672 shutdown_connections 2024-01-21T08:08:00.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.135+0000 7fcbe68ee700 1 --2- 172.21.15.78:0/3766453672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe00f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:00.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.135+0000 7fcbe68ee700 1 -- 172.21.15.78:0/3766453672 >> 172.21.15.78:0/3766453672 conn(0x7fcbe00f1c70 msgr2=0x7fcbe00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:00.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.135+0000 7fcbe68ee700 1 -- 172.21.15.78:0/3766453672 shutdown_connections 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.135+0000 7fcbe68ee700 1 -- 172.21.15.78:0/3766453672 wait complete. 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.136+0000 7fcbe68ee700 1 Processor -- start 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.136+0000 7fcbe68ee700 1 -- start start 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.136+0000 7fcbe68ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.136+0000 7fcbe68ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe0106c20 con 0x7fcbe00f6510 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.136+0000 7fcbdffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.136+0000 7fcbdffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:45320/0 (socket says 172.21.15.78:45320) 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.136+0000 7fcbdffff700 1 -- 172.21.15.78:0/2356670100 learned_addr learned my addr 172.21.15.78:0/2356670100 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.137+0000 7fcbdffff700 1 -- 172.21.15.78:0/2356670100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd0005040 con 0x7fcbe00f6510 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.137+0000 7fcbdffff700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe0106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fcbd0013510 tx=0x7fcbd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.137+0000 7fcbdcff9700 1 -- 172.21.15.78:0/2356670100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbd0009320 con 0x7fcbe00f6510 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.137+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe0106e20 con 0x7fcbe00f6510 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.137+0000 7fcbdcff9700 1 -- 172.21.15.78:0/2356670100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbd0013f00 con 0x7fcbe00f6510 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.137+0000 7fcbdcff9700 1 -- 172.21.15.78:0/2356670100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbd0014d90 con 0x7fcbe00f6510 2024-01-21T08:08:00.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.137+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe01072b0 con 0x7fcbe00f6510 2024-01-21T08:08:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.138+0000 7fcbdcff9700 1 -- 172.21.15.78:0/2356670100 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbd0014420 con 0x7fcbe00f6510 2024-01-21T08:08:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.138+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe0045ad0 con 0x7fcbe00f6510 2024-01-21T08:08:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.138+0000 7fcbdcff9700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcbc802ffb0 0x7fcbc8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.138+0000 7fcbdcff9700 1 -- 172.21.15.78:0/2356670100 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbd0040fe0 con 0x7fcbe00f6510 2024-01-21T08:08:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.138+0000 7fcbdf7fe700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcbc802ffb0 0x7fcbc8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.138+0000 7fcbdf7fe700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcbc802ffb0 0x7fcbc8032470 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fcbd40096a0 tx=0x7fcbd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.143+0000 7fcbdcff9700 1 -- 172.21.15.78:0/2356670100 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbd006d2b0 con 0x7fcbe00f6510 2024-01-21T08:08:00.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.349+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbe00fbf60 con 0x7fcbe00f6510 2024-01-21T08:08:00.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.350+0000 7fcbdcff9700 1 -- 172.21.15.78:0/2356670100 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcbd00095f0 con 0x7fcbe00f6510 2024-01-21T08:08:00.352 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:00.352 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:00.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcbc802ffb0 msgr2=0x7fcbc8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:00.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcbc802ffb0 0x7fcbc8032470 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fcbd40096a0 tx=0x7fcbd4006b40).stop 2024-01-21T08:08:00.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 msgr2=0x7fcbe0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe0106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fcbd0013510 tx=0x7fcbd000b040).stop 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 shutdown_connections 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcbc802ffb0 0x7fcbc8032470 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 --2- 172.21.15.78:0/2356670100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe00f6510 0x7fcbe0106730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 >> 172.21.15.78:0/2356670100 conn(0x7fcbe00f1c70 msgr2=0x7fcbe00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.351+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 shutdown_connections 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:00.352+0000 7fcbe68ee700 1 -- 172.21.15.78:0/2356670100 wait complete. 2024-01-21T08:08:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:07:58.614537+0000 mgr.smithi046.lmxnwc (mgr.14164) 84 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:00.350872+0000 mon.smithi046 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.78:0/2356670100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:01.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:01.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:00.614854+0000 mgr.smithi046.lmxnwc (mgr.14164) 85 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:02.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.990+0000 7f0700fa3700 1 -- 172.21.15.78:0/491554019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 msgr2=0x7f06fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.990+0000 7f0700fa3700 1 --2- 172.21.15.78:0/491554019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc0f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f06f0009fb0 tx=0x7f06f0009b30).stop 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.990+0000 7f0700fa3700 1 -- 172.21.15.78:0/491554019 shutdown_connections 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.990+0000 7f0700fa3700 1 --2- 172.21.15.78:0/491554019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc0f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.990+0000 7f0700fa3700 1 -- 172.21.15.78:0/491554019 >> 172.21.15.78:0/491554019 conn(0x7f06fc0f1c70 msgr2=0x7f06fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.990+0000 7f0700fa3700 1 -- 172.21.15.78:0/491554019 shutdown_connections 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.990+0000 7f0700fa3700 1 -- 172.21.15.78:0/491554019 wait complete. 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f0700fa3700 1 Processor -- start 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f0700fa3700 1 -- start start 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f0700fa3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f0700fa3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc106c20 con 0x7f06fc0f6510 2024-01-21T08:08:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f06fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f06fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35524/0 (socket says 172.21.15.78:35524) 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f06fa59c700 1 -- 172.21.15.78:0/2032751766 learned_addr learned my addr 172.21.15.78:0/2032751766 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.991+0000 7f06fa59c700 1 -- 172.21.15.78:0/2032751766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0005040 con 0x7f06fc0f6510 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.992+0000 7f06fa59c700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc106730 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f06f0013510 tx=0x7f06f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.992+0000 7f06eaffd700 1 -- 172.21.15.78:0/2032751766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f0009320 con 0x7f06fc0f6510 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.992+0000 7f06eaffd700 1 -- 172.21.15.78:0/2032751766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06f0013f00 con 0x7f06fc0f6510 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.992+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc106e20 con 0x7f06fc0f6510 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.992+0000 7f06eaffd700 1 -- 172.21.15.78:0/2032751766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f0014d90 con 0x7f06fc0f6510 2024-01-21T08:08:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.992+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06fc1072b0 con 0x7f06fc0f6510 2024-01-21T08:08:02.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.992+0000 7f06eaffd700 1 -- 172.21.15.78:0/2032751766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06f0014420 con 0x7f06fc0f6510 2024-01-21T08:08:02.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.993+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06fc045ad0 con 0x7f06fc0f6510 2024-01-21T08:08:02.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.993+0000 7f06eaffd700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06e402ff60 0x7f06e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:02.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.993+0000 7f06eaffd700 1 -- 172.21.15.78:0/2032751766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06f00410c0 con 0x7f06fc0f6510 2024-01-21T08:08:02.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.993+0000 7f06f9d9b700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06e402ff60 0x7f06e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:02.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.993+0000 7f06f9d9b700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06e402ff60 0x7f06e4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f06ec0096a0 tx=0x7f06ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:02.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:02.997+0000 7f06eaffd700 1 -- 172.21.15.78:0/2032751766 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06f00095f0 con 0x7f06fc0f6510 2024-01-21T08:08:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.203+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06fc0fbf60 con 0x7f06fc0f6510 2024-01-21T08:08:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.203+0000 7f06eaffd700 1 -- 172.21.15.78:0/2032751766 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06f001db70 con 0x7f06fc0f6510 2024-01-21T08:08:03.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:03.205 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06e402ff60 msgr2=0x7f06e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06e402ff60 0x7f06e4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f06ec0096a0 tx=0x7f06ec006b40).stop 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 msgr2=0x7f06fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc106730 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f06f0013510 tx=0x7f06f000b040).stop 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 shutdown_connections 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06e402ff60 0x7f06e4032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 --2- 172.21.15.78:0/2032751766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06fc0f6510 0x7f06fc106730 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 >> 172.21.15.78:0/2032751766 conn(0x7f06fc0f1c70 msgr2=0x7f06fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 shutdown_connections 2024-01-21T08:08:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:03.205+0000 7f0700fa3700 1 -- 172.21.15.78:0/2032751766 wait complete. 2024-01-21T08:08:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:08:02.281113+0000 mgr.smithi046.lmxnwc (mgr.14164) 86 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi046 ... 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Verifying port 9100 ... 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service failed because a timeout was exceeded. 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service" and "journalctl -xe" for details. 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:08:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: yield (conn, connr) 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: code, '\n'.join(err))) 2024-01-21T08:08:03.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi046 ... 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Verifying port 9100 ... 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service failed because a timeout was exceeded. 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service" and "journalctl -xe" for details. 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:08:03.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:08:02.288894+0000 mgr.smithi046.lmxnwc (mgr.14164) 87 : cephadm [INF] Deploying daemon prometheus.smithi046 on smithi046 2024-01-21T08:08:03.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:03.204308+0000 mon.smithi046 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.78:0/2032751766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:04.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:04.657 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:04.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:02.615074+0000 mgr.smithi046.lmxnwc (mgr.14164) 88 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.833+0000 7ff61cdb4700 1 -- 172.21.15.78:0/1609969288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 msgr2=0x7ff6180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.833+0000 7ff61cdb4700 1 --2- 172.21.15.78:0/1609969288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff6180f6930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff600009fb0 tx=0x7ff600009b30).stop 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.833+0000 7ff61cdb4700 1 -- 172.21.15.78:0/1609969288 shutdown_connections 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.833+0000 7ff61cdb4700 1 --2- 172.21.15.78:0/1609969288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff6180f6930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.833+0000 7ff61cdb4700 1 -- 172.21.15.78:0/1609969288 >> 172.21.15.78:0/1609969288 conn(0x7ff6180f1c70 msgr2=0x7ff6180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.833+0000 7ff61cdb4700 1 -- 172.21.15.78:0/1609969288 shutdown_connections 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.833+0000 7ff61cdb4700 1 -- 172.21.15.78:0/1609969288 wait complete. 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.834+0000 7ff61cdb4700 1 Processor -- start 2024-01-21T08:08:05.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.834+0000 7ff61cdb4700 1 -- start start 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.834+0000 7ff61cdb4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff61806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.834+0000 7ff61cdb4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618069e20 con 0x7ff6180f6510 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.834+0000 7ff61659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff61806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.834+0000 7ff61659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff61806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35536/0 (socket says 172.21.15.78:35536) 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.834+0000 7ff61659c700 1 -- 172.21.15.78:0/106764966 learned_addr learned my addr 172.21.15.78:0/106764966 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.835+0000 7ff61659c700 1 -- 172.21.15.78:0/106764966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600005040 con 0x7ff6180f6510 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.835+0000 7ff61659c700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff61806b9a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff600013510 tx=0x7ff60000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.835+0000 7ff60effd700 1 -- 172.21.15.78:0/106764966 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff600009320 con 0x7ff6180f6510 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.835+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff618069fc0 con 0x7ff6180f6510 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.835+0000 7ff60effd700 1 -- 172.21.15.78:0/106764966 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff600013f00 con 0x7ff6180f6510 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.835+0000 7ff60effd700 1 -- 172.21.15.78:0/106764966 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff600014d90 con 0x7ff6180f6510 2024-01-21T08:08:05.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.835+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61806a450 con 0x7ff6180f6510 2024-01-21T08:08:05.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.836+0000 7ff60effd700 1 -- 172.21.15.78:0/106764966 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff600014420 con 0x7ff6180f6510 2024-01-21T08:08:05.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.836+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff618059100 con 0x7ff6180f6510 2024-01-21T08:08:05.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.836+0000 7ff60effd700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff60402ff60 0x7ff604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:05.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.836+0000 7ff60effd700 1 -- 172.21.15.78:0/106764966 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff600040fb0 con 0x7ff6180f6510 2024-01-21T08:08:05.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.836+0000 7ff615d9b700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff60402ff60 0x7ff604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:05.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.837+0000 7ff615d9b700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff60402ff60 0x7ff604032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff6080096a0 tx=0x7ff608006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:05.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:05.841+0000 7ff60effd700 1 -- 172.21.15.78:0/106764966 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6000095f0 con 0x7ff6180f6510 2024-01-21T08:08:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.049+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6180fbf60 con 0x7ff6180f6510 2024-01-21T08:08:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.049+0000 7ff60effd700 1 -- 172.21.15.78:0/106764966 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff60001db70 con 0x7ff6180f6510 2024-01-21T08:08:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:06.051 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff60402ff60 msgr2=0x7ff604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff60402ff60 0x7ff604032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff6080096a0 tx=0x7ff608006b40).stop 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 msgr2=0x7ff61806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff61806b9a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff600013510 tx=0x7ff60000b040).stop 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 shutdown_connections 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff60402ff60 0x7ff604032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 --2- 172.21.15.78:0/106764966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180f6510 0x7ff61806b9a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 >> 172.21.15.78:0/106764966 conn(0x7ff6180f1c70 msgr2=0x7ff6180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 shutdown_connections 2024-01-21T08:08:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:06.051+0000 7ff61cdb4700 1 -- 172.21.15.78:0/106764966 wait complete. 2024-01-21T08:08:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:04.615419+0000 mgr.smithi046.lmxnwc (mgr.14164) 89 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:06.050449+0000 mon.smithi046 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.78:0/106764966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:07.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:07.447 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/2577429712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 msgr2=0x7f7dc00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 --2- 172.21.15.78:0/2577429712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc00f55c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7db4009fb0 tx=0x7f7db4009b30).stop 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/2577429712 shutdown_connections 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 --2- 172.21.15.78:0/2577429712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc00f55c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/2577429712 >> 172.21.15.78:0/2577429712 conn(0x7f7dc00f09a0 msgr2=0x7f7dc00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/2577429712 shutdown_connections 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/2577429712 wait complete. 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 Processor -- start 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.641+0000 7f7dc5b3c700 1 -- start start 2024-01-21T08:08:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.642+0000 7f7dc5b3c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.642+0000 7f7dc5b3c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc0060240 con 0x7f7dc00f51a0 2024-01-21T08:08:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.642+0000 7f7dc4b3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.642+0000 7f7dc4b3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35556/0 (socket says 172.21.15.78:35556) 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.642+0000 7f7dc4b3a700 1 -- 172.21.15.78:0/842537705 learned_addr learned my addr 172.21.15.78:0/842537705 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.643+0000 7f7dc4b3a700 1 -- 172.21.15.78:0/842537705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4005040 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.643+0000 7f7dc4b3a700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc01091b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7db4013510 tx=0x7f7db400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.643+0000 7f7dbd7fa700 1 -- 172.21.15.78:0/842537705 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db4009320 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.643+0000 7f7dbd7fa700 1 -- 172.21.15.78:0/842537705 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7db4013f00 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.643+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc0060440 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.643+0000 7f7dbd7fa700 1 -- 172.21.15.78:0/842537705 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db4014dc0 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.643+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc0109860 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.644+0000 7f7dbd7fa700 1 -- 172.21.15.78:0/842537705 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7db40095f0 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.644+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc0045ad0 con 0x7f7dc00f51a0 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.644+0000 7f7dbd7fa700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7db002ff60 0x7f7db0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.644+0000 7f7dbd7fa700 1 -- 172.21.15.78:0/842537705 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7db4041040 con 0x7f7dc00f51a0 2024-01-21T08:08:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.644+0000 7f7dbffff700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7db002ff60 0x7f7db0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.644+0000 7f7dbffff700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7db002ff60 0x7f7db0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7dac0096a0 tx=0x7f7dac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.649+0000 7f7dbd7fa700 1 -- 172.21.15.78:0/842537705 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7db4014420 con 0x7f7dc00f51a0 2024-01-21T08:08:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:06.615789+0000 mgr.smithi046.lmxnwc (mgr.14164) 90 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.854+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dc010a080 con 0x7f7dc00f51a0 2024-01-21T08:08:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.855+0000 7f7dbd7fa700 1 -- 172.21.15.78:0/842537705 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7db4010230 con 0x7f7dc00f51a0 2024-01-21T08:08:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:08.857 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:08.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.856+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7db002ff60 msgr2=0x7f7db0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:08.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.856+0000 7f7dc5b3c700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7db002ff60 0x7f7db0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7dac0096a0 tx=0x7f7dac006b40).stop 2024-01-21T08:08:08.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.856+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 msgr2=0x7f7dc01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:08.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.856+0000 7f7dc5b3c700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc01091b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7db4013510 tx=0x7f7db400b040).stop 2024-01-21T08:08:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.857+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 shutdown_connections 2024-01-21T08:08:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.857+0000 7f7dc5b3c700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f7db002ff60 0x7f7db0032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.857+0000 7f7dc5b3c700 1 --2- 172.21.15.78:0/842537705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00f51a0 0x7f7dc01091b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.857+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 >> 172.21.15.78:0/842537705 conn(0x7f7dc00f09a0 msgr2=0x7f7dc00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.857+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 shutdown_connections 2024-01-21T08:08:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:08.857+0000 7f7dc5b3c700 1 -- 172.21.15.78:0/842537705 wait complete. 2024-01-21T08:08:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:09.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:08.856049+0000 mon.smithi046 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.78:0/842537705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:10.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:10.266 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:08.616149+0000 mgr.smithi046.lmxnwc (mgr.14164) 91 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:11.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 -- 172.21.15.78:0/21882635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 msgr2=0x7f06700f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 --2- 172.21.15.78:0/21882635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f9170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0660009fb0 tx=0x7f0660009b30).stop 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 -- 172.21.15.78:0/21882635 shutdown_connections 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 --2- 172.21.15.78:0/21882635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f9170 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 -- 172.21.15.78:0/21882635 >> 172.21.15.78:0/21882635 conn(0x7f06700f0b70 msgr2=0x7f06700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 -- 172.21.15.78:0/21882635 shutdown_connections 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 -- 172.21.15.78:0/21882635 wait complete. 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.467+0000 7f0676654700 1 Processor -- start 2024-01-21T08:08:11.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.468+0000 7f0676654700 1 -- start start 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.468+0000 7f0676654700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.468+0000 7f0676654700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06700f5070 con 0x7f0670060200 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.468+0000 7f066ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.468+0000 7f066ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35572/0 (socket says 172.21.15.78:35572) 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.468+0000 7f066ffff700 1 -- 172.21.15.78:0/2579865975 learned_addr learned my addr 172.21.15.78:0/2579865975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.469+0000 7f066ffff700 1 -- 172.21.15.78:0/2579865975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0660005040 con 0x7f0670060200 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.469+0000 7f066ffff700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f6bf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0660013510 tx=0x7f066000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.469+0000 7f066cff9700 1 -- 172.21.15.78:0/2579865975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0660009320 con 0x7f0670060200 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.469+0000 7f066cff9700 1 -- 172.21.15.78:0/2579865975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0660013f00 con 0x7f0670060200 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.469+0000 7f066cff9700 1 -- 172.21.15.78:0/2579865975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0660014dc0 con 0x7f0670060200 2024-01-21T08:08:11.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.469+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06700f5210 con 0x7f0670060200 2024-01-21T08:08:11.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.469+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06700f56a0 con 0x7f0670060200 2024-01-21T08:08:11.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.470+0000 7f066cff9700 1 -- 172.21.15.78:0/2579865975 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0660014420 con 0x7f0670060200 2024-01-21T08:08:11.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.470+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0670059100 con 0x7f0670060200 2024-01-21T08:08:11.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.470+0000 7f066cff9700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f065c02ff60 0x7f065c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:11.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.470+0000 7f066cff9700 1 -- 172.21.15.78:0/2579865975 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0660041030 con 0x7f0670060200 2024-01-21T08:08:11.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.470+0000 7f066f7fe700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f065c02ff60 0x7f065c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:11.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.471+0000 7f066f7fe700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f065c02ff60 0x7f065c032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f06580096a0 tx=0x7f0658006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:11.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.475+0000 7f066cff9700 1 -- 172.21.15.78:0/2579865975 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06600095f0 con 0x7f0670060200 2024-01-21T08:08:11.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.681+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06700fbf50 con 0x7f0670060200 2024-01-21T08:08:11.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.682+0000 7f066cff9700 1 -- 172.21.15.78:0/2579865975 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f066001cb10 con 0x7f0670060200 2024-01-21T08:08:11.684 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:11.684 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:11.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.683+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f065c02ff60 msgr2=0x7f065c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.683+0000 7f0676654700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f065c02ff60 0x7f065c032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f06580096a0 tx=0x7f0658006b40).stop 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.683+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 msgr2=0x7f06700f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.684+0000 7f0676654700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f6bf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0660013510 tx=0x7f066000b040).stop 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.684+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 shutdown_connections 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.684+0000 7f0676654700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f065c02ff60 0x7f065c032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.684+0000 7f0676654700 1 --2- 172.21.15.78:0/2579865975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0670060200 0x7f06700f6bf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.684+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 >> 172.21.15.78:0/2579865975 conn(0x7f06700f0b70 msgr2=0x7f06700f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.684+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 shutdown_connections 2024-01-21T08:08:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:11.684+0000 7f0676654700 1 -- 172.21.15.78:0/2579865975 wait complete. 2024-01-21T08:08:11.687 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:12.625 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:10.616507+0000 mgr.smithi046.lmxnwc (mgr.14164) 92 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:12.625 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:11.683053+0000 mon.smithi046 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.78:0/2579865975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:13.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:13.089 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.350+0000 7f1b465ad700 1 -- 172.21.15.78:0/780704610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 msgr2=0x7f1b400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.350+0000 7f1b465ad700 1 --2- 172.21.15.78:0/780704610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b400f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1b30009fb0 tx=0x7f1b30009b30).stop 2024-01-21T08:08:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.350+0000 7f1b465ad700 1 -- 172.21.15.78:0/780704610 shutdown_connections 2024-01-21T08:08:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.350+0000 7f1b465ad700 1 --2- 172.21.15.78:0/780704610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b400f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.350+0000 7f1b465ad700 1 -- 172.21.15.78:0/780704610 >> 172.21.15.78:0/780704610 conn(0x7f1b400f0b70 msgr2=0x7f1b400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.350+0000 7f1b465ad700 1 -- 172.21.15.78:0/780704610 shutdown_connections 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.350+0000 7f1b465ad700 1 -- 172.21.15.78:0/780704610 wait complete. 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.351+0000 7f1b465ad700 1 Processor -- start 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.351+0000 7f1b465ad700 1 -- start start 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.351+0000 7f1b465ad700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b40108940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.351+0000 7f1b465ad700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b40108e30 con 0x7f1b400f5410 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.351+0000 7f1b3ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b40108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.351+0000 7f1b3ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b40108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:50906/0 (socket says 172.21.15.78:50906) 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.351+0000 7f1b3ffff700 1 -- 172.21.15.78:0/191041241 learned_addr learned my addr 172.21.15.78:0/191041241 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.352+0000 7f1b3ffff700 1 -- 172.21.15.78:0/191041241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30005040 con 0x7f1b400f5410 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.352+0000 7f1b3ffff700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b40108940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1b30013510 tx=0x7f1b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.352+0000 7f1b3cff9700 1 -- 172.21.15.78:0/191041241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b30009320 con 0x7f1b400f5410 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.352+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b40109030 con 0x7f1b400f5410 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.352+0000 7f1b3cff9700 1 -- 172.21.15.78:0/191041241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b30013f00 con 0x7f1b400f5410 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.352+0000 7f1b3cff9700 1 -- 172.21.15.78:0/191041241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b30014d90 con 0x7f1b400f5410 2024-01-21T08:08:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.352+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b401094c0 con 0x7f1b400f5410 2024-01-21T08:08:14.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.353+0000 7f1b3cff9700 1 -- 172.21.15.78:0/191041241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b300095f0 con 0x7f1b400f5410 2024-01-21T08:08:14.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.353+0000 7f1b3cff9700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b2c02ff60 0x7f1b2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.353+0000 7f1b3cff9700 1 -- 172.21.15.78:0/191041241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b30040fe0 con 0x7f1b400f5410 2024-01-21T08:08:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.353+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b40045ad0 con 0x7f1b400f5410 2024-01-21T08:08:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.353+0000 7f1b3f7fe700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b2c02ff60 0x7f1b2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.353+0000 7f1b3f7fe700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b2c02ff60 0x7f1b2c032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1b280096a0 tx=0x7f1b28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.358+0000 7f1b3cff9700 1 -- 172.21.15.78:0/191041241 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b3001d7f0 con 0x7f1b400f5410 2024-01-21T08:08:14.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.560+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b400faf50 con 0x7f1b400f5410 2024-01-21T08:08:14.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.561+0000 7f1b3cff9700 1 -- 172.21.15.78:0/191041241 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b3002b020 con 0x7f1b400f5410 2024-01-21T08:08:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:14.563 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b2c02ff60 msgr2=0x7f1b2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b2c02ff60 0x7f1b2c032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1b280096a0 tx=0x7f1b28006b40).stop 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 msgr2=0x7f1b40108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b40108940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1b30013510 tx=0x7f1b3000b040).stop 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 shutdown_connections 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1b2c02ff60 0x7f1b2c032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 --2- 172.21.15.78:0/191041241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b400f5410 0x7f1b40108940 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 >> 172.21.15.78:0/191041241 conn(0x7f1b400f0b70 msgr2=0x7f1b400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 shutdown_connections 2024-01-21T08:08:14.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:14.563+0000 7f1b465ad700 1 -- 172.21.15.78:0/191041241 wait complete. 2024-01-21T08:08:14.566 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:14.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:12.616769+0000 mgr.smithi046.lmxnwc (mgr.14164) 93 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:14.562079+0000 mon.smithi046 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.78:0/191041241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:16.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:16.003 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:14.617103+0000 mgr.smithi046.lmxnwc (mgr.14164) 94 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 -- 172.21.15.78:0/779416049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 msgr2=0x7f33980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 --2- 172.21.15.78:0/779416049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f3388009fb0 tx=0x7f3388009b30).stop 2024-01-21T08:08:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 -- 172.21.15.78:0/779416049 shutdown_connections 2024-01-21T08:08:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 --2- 172.21.15.78:0/779416049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 -- 172.21.15.78:0/779416049 >> 172.21.15.78:0/779416049 conn(0x7f33980f0b70 msgr2=0x7f33980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 -- 172.21.15.78:0/779416049 shutdown_connections 2024-01-21T08:08:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 -- 172.21.15.78:0/779416049 wait complete. 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.126+0000 7f339e2c4700 1 Processor -- start 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f339e2c4700 1 -- start start 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f339e2c4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f339e2c4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33980fb820 con 0x7f33980f5410 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f3397fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f3397fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:50918/0 (socket says 172.21.15.78:50918) 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f3397fff700 1 -- 172.21.15.78:0/681641729 learned_addr learned my addr 172.21.15.78:0/681641729 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f3397fff700 1 -- 172.21.15.78:0/681641729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3388005040 con 0x7f33980f5410 2024-01-21T08:08:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.127+0000 7f3397fff700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980fd3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f3388013510 tx=0x7f338800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.128+0000 7f3394ff9700 1 -- 172.21.15.78:0/681641729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3388009320 con 0x7f33980f5410 2024-01-21T08:08:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.128+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33980fb9c0 con 0x7f33980f5410 2024-01-21T08:08:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.128+0000 7f3394ff9700 1 -- 172.21.15.78:0/681641729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3388013f00 con 0x7f33980f5410 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.128+0000 7f3394ff9700 1 -- 172.21.15.78:0/681641729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3388014d90 con 0x7f33980f5410 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.128+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33980fbe50 con 0x7f33980f5410 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.128+0000 7f3394ff9700 1 -- 172.21.15.78:0/681641729 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33880263a0 con 0x7f33980f5410 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.129+0000 7f3394ff9700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f338002ff60 0x7f3380032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.129+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3398059100 con 0x7f33980f5410 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.129+0000 7f3394ff9700 1 -- 172.21.15.78:0/681641729 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3388041020 con 0x7f33980f5410 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.129+0000 7f33977fe700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f338002ff60 0x7f3380032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.129+0000 7f33977fe700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f338002ff60 0x7f3380032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f338c0096a0 tx=0x7f338c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:17.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.133+0000 7f3394ff9700 1 -- 172.21.15.78:0/681641729 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3388012920 con 0x7f33980f5410 2024-01-21T08:08:17.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.339+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33980fae60 con 0x7f33980f5410 2024-01-21T08:08:17.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.340+0000 7f3394ff9700 1 -- 172.21.15.78:0/681641729 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f338802b020 con 0x7f33980f5410 2024-01-21T08:08:17.341 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:17.342 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:17.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f338002ff60 msgr2=0x7f3380032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:17.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f338002ff60 0x7f3380032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f338c0096a0 tx=0x7f338c006b40).stop 2024-01-21T08:08:17.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 msgr2=0x7f33980fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:17.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980fd3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f3388013510 tx=0x7f338800b040).stop 2024-01-21T08:08:17.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 shutdown_connections 2024-01-21T08:08:17.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f338002ff60 0x7f3380032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:17.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 --2- 172.21.15.78:0/681641729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33980f5410 0x7f33980fd3a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:17.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 >> 172.21.15.78:0/681641729 conn(0x7f33980f0b70 msgr2=0x7f339805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:17.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 shutdown_connections 2024-01-21T08:08:17.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:17.341+0000 7f339e2c4700 1 -- 172.21.15.78:0/681641729 wait complete. 2024-01-21T08:08:17.344 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:18.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:18.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:18.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:16.617417+0000 mgr.smithi046.lmxnwc (mgr.14164) 95 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:18.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:17.340755+0000 mon.smithi046 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.78:0/681641729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:19.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 -- 172.21.15.78:0/2771860945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 msgr2=0x7ff77c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 --2- 172.21.15.78:0/2771860945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff76c009fb0 tx=0x7ff76c009b30).stop 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 -- 172.21.15.78:0/2771860945 shutdown_connections 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 --2- 172.21.15.78:0/2771860945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 -- 172.21.15.78:0/2771860945 >> 172.21.15.78:0/2771860945 conn(0x7ff77c0f0b70 msgr2=0x7ff77c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 -- 172.21.15.78:0/2771860945 shutdown_connections 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 -- 172.21.15.78:0/2771860945 wait complete. 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 Processor -- start 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 -- start start 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.924+0000 7ff781a0d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0fb820 con 0x7ff77c0f5410 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff77affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:19.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff77affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:50940/0 (socket says 172.21.15.78:50940) 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff77affd700 1 -- 172.21.15.78:0/2870904588 learned_addr learned my addr 172.21.15.78:0/2870904588 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff77affd700 1 -- 172.21.15.78:0/2870904588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c005040 con 0x7ff77c0f5410 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff77affd700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0fd3a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff76c014000 tx=0x7ff76c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff7637fe700 1 -- 172.21.15.78:0/2870904588 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff76c0095f0 con 0x7ff77c0f5410 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff7637fe700 1 -- 172.21.15.78:0/2870904588 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff76c007e70 con 0x7ff77c0f5410 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c0fb9c0 con 0x7ff77c0f5410 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff7637fe700 1 -- 172.21.15.78:0/2870904588 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff76c014ea0 con 0x7ff77c0f5410 2024-01-21T08:08:19.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.925+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c0fbe50 con 0x7ff77c0f5410 2024-01-21T08:08:19.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.926+0000 7ff7637fe700 1 -- 172.21.15.78:0/2870904588 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff76c025070 con 0x7ff77c0f5410 2024-01-21T08:08:19.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.926+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c059100 con 0x7ff77c0f5410 2024-01-21T08:08:19.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.926+0000 7ff7637fe700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff76402ff60 0x7ff764032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:19.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.926+0000 7ff7637fe700 1 -- 172.21.15.78:0/2870904588 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff76c040930 con 0x7ff77c0f5410 2024-01-21T08:08:19.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.926+0000 7ff77a7fc700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff76402ff60 0x7ff764032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:19.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.927+0000 7ff77a7fc700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff76402ff60 0x7ff764032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff7700096a0 tx=0x7ff770006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:19.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:19.931+0000 7ff7637fe700 1 -- 172.21.15.78:0/2870904588 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff76c00f4e0 con 0x7ff77c0f5410 2024-01-21T08:08:20.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.137+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff77c0fae60 con 0x7ff77c0f5410 2024-01-21T08:08:20.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.138+0000 7ff7637fe700 1 -- 172.21.15.78:0/2870904588 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff76c00f300 con 0x7ff77c0f5410 2024-01-21T08:08:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff76402ff60 msgr2=0x7ff764032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff76402ff60 0x7ff764032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff7700096a0 tx=0x7ff770006b40).stop 2024-01-21T08:08:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 msgr2=0x7ff77c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0fd3a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff76c014000 tx=0x7ff76c00b040).stop 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 shutdown_connections 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff76402ff60 0x7ff764032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 --2- 172.21.15.78:0/2870904588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff77c0f5410 0x7ff77c0fd3a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 >> 172.21.15.78:0/2870904588 conn(0x7ff77c0f0b70 msgr2=0x7ff77c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 shutdown_connections 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:20.140+0000 7ff781a0d700 1 -- 172.21.15.78:0/2870904588 wait complete. 2024-01-21T08:08:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:18.617726+0000 mgr.smithi046.lmxnwc (mgr.14164) 96 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:20.139428+0000 mon.smithi046 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.78:0/2870904588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:21.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:21.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:20.741056+0000 mon.smithi046 (mon.0) 185 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:08:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:20.745054+0000 mon.smithi046 (mon.0) 186 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:08:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.727+0000 7f0714897700 1 -- 172.21.15.78:0/4112517806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 msgr2=0x7f07100f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.727+0000 7f0714897700 1 --2- 172.21.15.78:0/4112517806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f07100f4be0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f0700009fb0 tx=0x7f0700009b30).stop 2024-01-21T08:08:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.727+0000 7f0714897700 1 -- 172.21.15.78:0/4112517806 shutdown_connections 2024-01-21T08:08:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.727+0000 7f0714897700 1 --2- 172.21.15.78:0/4112517806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f07100f4be0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.727+0000 7f0714897700 1 -- 172.21.15.78:0/4112517806 >> 172.21.15.78:0/4112517806 conn(0x7f07100f02e0 msgr2=0x7f07100f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.727+0000 7f0714897700 1 -- 172.21.15.78:0/4112517806 shutdown_connections 2024-01-21T08:08:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.727+0000 7f0714897700 1 -- 172.21.15.78:0/4112517806 wait complete. 2024-01-21T08:08:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.728+0000 7f0714897700 1 Processor -- start 2024-01-21T08:08:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.728+0000 7f0714897700 1 -- start start 2024-01-21T08:08:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.728+0000 7f0714897700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f0710106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.728+0000 7f0714897700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0710106bf0 con 0x7f07100f47c0 2024-01-21T08:08:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.728+0000 7f070e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f0710106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.728+0000 7f070e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f0710106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49918/0 (socket says 172.21.15.78:49918) 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.728+0000 7f070e59c700 1 -- 172.21.15.78:0/3272298247 learned_addr learned my addr 172.21.15.78:0/3272298247 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.729+0000 7f070e59c700 1 -- 172.21.15.78:0/3272298247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700005040 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.729+0000 7f070e59c700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f0710106700 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0700013510 tx=0x7f070000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.729+0000 7f06feffd700 1 -- 172.21.15.78:0/3272298247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0700009320 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.729+0000 7f06feffd700 1 -- 172.21.15.78:0/3272298247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0700013f00 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.729+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0710106df0 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.729+0000 7f06feffd700 1 -- 172.21.15.78:0/3272298247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0700014d90 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.729+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0710107280 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.730+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0710045ad0 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.730+0000 7f06feffd700 1 -- 172.21.15.78:0/3272298247 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07000095f0 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.730+0000 7f06feffd700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06f802ff60 0x7f06f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.730+0000 7f06feffd700 1 -- 172.21.15.78:0/3272298247 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0700041000 con 0x7f07100f47c0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.730+0000 7f070dd9b700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06f802ff60 0x7f06f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.731+0000 7f070dd9b700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06f802ff60 0x7f06f8032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f07040096a0 tx=0x7f0704006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:22.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.735+0000 7f06feffd700 1 -- 172.21.15.78:0/3272298247 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0700014420 con 0x7f07100f47c0 2024-01-21T08:08:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:20.618006+0000 mgr.smithi046.lmxnwc (mgr.14164) 97 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:22.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.943+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0710108130 con 0x7f07100f47c0 2024-01-21T08:08:22.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.944+0000 7f06feffd700 1 -- 172.21.15.78:0/3272298247 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0700010230 con 0x7f07100f47c0 2024-01-21T08:08:22.946 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:22.946 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:22.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06f802ff60 msgr2=0x7f06f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:22.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06f802ff60 0x7f06f8032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f07040096a0 tx=0x7f0704006b40).stop 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 msgr2=0x7f0710106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f0710106700 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0700013510 tx=0x7f070000b040).stop 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 shutdown_connections 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f06f802ff60 0x7f06f8032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 --2- 172.21.15.78:0/3272298247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07100f47c0 0x7f0710106700 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.945+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 >> 172.21.15.78:0/3272298247 conn(0x7f07100f02e0 msgr2=0x7f07100f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.946+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 shutdown_connections 2024-01-21T08:08:22.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:22.946+0000 7f0714897700 1 -- 172.21.15.78:0/3272298247 wait complete. 2024-01-21T08:08:22.949 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:22.944972+0000 mon.smithi046 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.78:0/3272298247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:24.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:24.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:22.618386+0000 mgr.smithi046.lmxnwc (mgr.14164) 98 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:25.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 -- 172.21.15.78:0/597563004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 msgr2=0x7fa6100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:25.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 --2- 172.21.15.78:0/597563004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6100f90f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa60c009fb0 tx=0x7fa60c009b30).stop 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 -- 172.21.15.78:0/597563004 shutdown_connections 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 --2- 172.21.15.78:0/597563004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6100f90f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 -- 172.21.15.78:0/597563004 >> 172.21.15.78:0/597563004 conn(0x7fa6100f0b70 msgr2=0x7fa6100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 -- 172.21.15.78:0/597563004 shutdown_connections 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 -- 172.21.15.78:0/597563004 wait complete. 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 Processor -- start 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.565+0000 7fa618639700 1 -- start start 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.566+0000 7fa618639700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.566+0000 7fa618639700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa610106bd0 con 0x7fa610060280 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.566+0000 7fa6163d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.566+0000 7fa6163d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49946/0 (socket says 172.21.15.78:49946) 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.566+0000 7fa6163d5700 1 -- 172.21.15.78:0/2924992354 learned_addr learned my addr 172.21.15.78:0/2924992354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.566+0000 7fa6163d5700 1 -- 172.21.15.78:0/2924992354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c005040 con 0x7fa610060280 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.566+0000 7fa6163d5700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6101066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa60c013510 tx=0x7fa60c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.567+0000 7fa606ffd700 1 -- 172.21.15.78:0/2924992354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa60c009320 con 0x7fa610060280 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.567+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa610106dd0 con 0x7fa610060280 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.567+0000 7fa606ffd700 1 -- 172.21.15.78:0/2924992354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa60c013f00 con 0x7fa610060280 2024-01-21T08:08:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.567+0000 7fa606ffd700 1 -- 172.21.15.78:0/2924992354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa60c014dc0 con 0x7fa610060280 2024-01-21T08:08:25.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.567+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa610107260 con 0x7fa610060280 2024-01-21T08:08:25.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.568+0000 7fa606ffd700 1 -- 172.21.15.78:0/2924992354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa60c0095f0 con 0x7fa610060280 2024-01-21T08:08:25.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.568+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa610045ad0 con 0x7fa610060280 2024-01-21T08:08:25.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.568+0000 7fa606ffd700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa5fc02ff60 0x7fa5fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:25.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.568+0000 7fa606ffd700 1 -- 172.21.15.78:0/2924992354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa60c041000 con 0x7fa610060280 2024-01-21T08:08:25.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.568+0000 7fa615bd4700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa5fc02ff60 0x7fa5fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:25.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.568+0000 7fa615bd4700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa5fc02ff60 0x7fa5fc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa6000096a0 tx=0x7fa600006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:25.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.573+0000 7fa606ffd700 1 -- 172.21.15.78:0/2924992354 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa60c01c790 con 0x7fa610060280 2024-01-21T08:08:25.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.776+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa610108110 con 0x7fa610060280 2024-01-21T08:08:25.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.777+0000 7fa606ffd700 1 -- 172.21.15.78:0/2924992354 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa60c010230 con 0x7fa610060280 2024-01-21T08:08:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:25.779 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:25.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa5fc02ff60 msgr2=0x7fa5fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa5fc02ff60 0x7fa5fc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa6000096a0 tx=0x7fa600006b40).stop 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 msgr2=0x7fa6101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6101066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa60c013510 tx=0x7fa60c00b040).stop 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 shutdown_connections 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa5fc02ff60 0x7fa5fc032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 --2- 172.21.15.78:0/2924992354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa610060280 0x7fa6101066e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 >> 172.21.15.78:0/2924992354 conn(0x7fa6100f0b70 msgr2=0x7fa6100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 shutdown_connections 2024-01-21T08:08:25.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:25.779+0000 7fa618639700 1 -- 172.21.15.78:0/2924992354 wait complete. 2024-01-21T08:08:25.782 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:24.618658+0000 mgr.smithi046.lmxnwc (mgr.14164) 99 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:25.778245+0000 mon.smithi046 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.78:0/2924992354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:27.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:27.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.305+0000 7f67ad4c0700 1 -- 172.21.15.78:0/1451429924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 msgr2=0x7f67a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.305+0000 7f67ad4c0700 1 --2- 172.21.15.78:0/1451429924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f6790009fb0 tx=0x7f6790009b30).stop 2024-01-21T08:08:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.305+0000 7f67ad4c0700 1 -- 172.21.15.78:0/1451429924 shutdown_connections 2024-01-21T08:08:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.305+0000 7f67ad4c0700 1 --2- 172.21.15.78:0/1451429924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.305+0000 7f67ad4c0700 1 -- 172.21.15.78:0/1451429924 >> 172.21.15.78:0/1451429924 conn(0x7f67a80f0b70 msgr2=0x7f67a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.306+0000 7f67ad4c0700 1 -- 172.21.15.78:0/1451429924 shutdown_connections 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.306+0000 7f67ad4c0700 1 -- 172.21.15.78:0/1451429924 wait complete. 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.306+0000 7f67ad4c0700 1 Processor -- start 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.306+0000 7f67ad4c0700 1 -- start start 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.306+0000 7f67ad4c0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.306+0000 7f67ad4c0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a80fb820 con 0x7f67a80f5410 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f67a6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f67a6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49964/0 (socket says 172.21.15.78:49964) 2024-01-21T08:08:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f67a6ffd700 1 -- 172.21.15.78:0/2219322910 learned_addr learned my addr 172.21.15.78:0/2219322910 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f67a6ffd700 1 -- 172.21.15.78:0/2219322910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6790005040 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f67a6ffd700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6790013510 tx=0x7f679000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f679f7fe700 1 -- 172.21.15.78:0/2219322910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6790009320 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a80fb9c0 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f679f7fe700 1 -- 172.21.15.78:0/2219322910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6790013f00 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f679f7fe700 1 -- 172.21.15.78:0/2219322910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6790014d90 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.307+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a80fbe50 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.308+0000 7f679f7fe700 1 -- 172.21.15.78:0/2219322910 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67900095f0 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.308+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a8059100 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.308+0000 7f679f7fe700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f679402ff60 0x7f6794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.308+0000 7f679f7fe700 1 -- 172.21.15.78:0/2219322910 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67900410f0 con 0x7f67a80f5410 2024-01-21T08:08:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.308+0000 7f67a67fc700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f679402ff60 0x7f6794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.309+0000 7f67a67fc700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f679402ff60 0x7f6794032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f67980096a0 tx=0x7f6798006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.313+0000 7f679f7fe700 1 -- 172.21.15.78:0/2219322910 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67900268c0 con 0x7f67a80f5410 2024-01-21T08:08:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.521+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67a80fae60 con 0x7f67a80f5410 2024-01-21T08:08:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.521+0000 7f679f7fe700 1 -- 172.21.15.78:0/2219322910 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6790010230 con 0x7f67a80f5410 2024-01-21T08:08:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:28.523 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f679402ff60 msgr2=0x7f6794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f679402ff60 0x7f6794032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f67980096a0 tx=0x7f6798006b40).stop 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 msgr2=0x7f67a80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6790013510 tx=0x7f679000b040).stop 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 shutdown_connections 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f679402ff60 0x7f6794032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 --2- 172.21.15.78:0/2219322910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67a80f5410 0x7f67a80fd3a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 >> 172.21.15.78:0/2219322910 conn(0x7f67a80f0b70 msgr2=0x7f67a805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 shutdown_connections 2024-01-21T08:08:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:28.523+0000 7f67ad4c0700 1 -- 172.21.15.78:0/2219322910 wait complete. 2024-01-21T08:08:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:26.618990+0000 mgr.smithi046.lmxnwc (mgr.14164) 100 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:29.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:28.522448+0000 mon.smithi046 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.78:0/2219322910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:29.927 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:29.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:28.619383+0000 mgr.smithi046.lmxnwc (mgr.14164) 101 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:31.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.141+0000 7f34178ae700 1 -- 172.21.15.78:0/2634150256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 msgr2=0x7f34100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:31.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.141+0000 7f34178ae700 1 --2- 172.21.15.78:0/2634150256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34100f90f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3400009fb0 tx=0x7f3400009b30).stop 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.141+0000 7f34178ae700 1 -- 172.21.15.78:0/2634150256 shutdown_connections 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.141+0000 7f34178ae700 1 --2- 172.21.15.78:0/2634150256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34100f90f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.141+0000 7f34178ae700 1 -- 172.21.15.78:0/2634150256 >> 172.21.15.78:0/2634150256 conn(0x7f34100f0b70 msgr2=0x7f34100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.141+0000 7f34178ae700 1 -- 172.21.15.78:0/2634150256 shutdown_connections 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.141+0000 7f34178ae700 1 -- 172.21.15.78:0/2634150256 wait complete. 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.142+0000 7f34178ae700 1 Processor -- start 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.142+0000 7f34178ae700 1 -- start start 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.142+0000 7f34178ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.142+0000 7f34178ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3410106bd0 con 0x7f3410060280 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f341564a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f341564a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49986/0 (socket says 172.21.15.78:49986) 2024-01-21T08:08:31.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f341564a700 1 -- 172.21.15.78:0/3658617505 learned_addr learned my addr 172.21.15.78:0/3658617505 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:31.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f341564a700 1 -- 172.21.15.78:0/3658617505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400005040 con 0x7f3410060280 2024-01-21T08:08:31.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f341564a700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34101066e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f3400013510 tx=0x7f340000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:31.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f3405ffb700 1 -- 172.21.15.78:0/3658617505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3400009320 con 0x7f3410060280 2024-01-21T08:08:31.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f3405ffb700 1 -- 172.21.15.78:0/3658617505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3400013f00 con 0x7f3410060280 2024-01-21T08:08:31.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3410106dd0 con 0x7f3410060280 2024-01-21T08:08:31.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f3405ffb700 1 -- 172.21.15.78:0/3658617505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3400014dc0 con 0x7f3410060280 2024-01-21T08:08:31.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.143+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3410107260 con 0x7f3410060280 2024-01-21T08:08:31.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.144+0000 7f3405ffb700 1 -- 172.21.15.78:0/3658617505 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3400014420 con 0x7f3410060280 2024-01-21T08:08:31.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.144+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3410045ad0 con 0x7f3410060280 2024-01-21T08:08:31.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.144+0000 7f3405ffb700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f33fc02ff60 0x7f33fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:31.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.144+0000 7f3405ffb700 1 -- 172.21.15.78:0/3658617505 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34000410f0 con 0x7f3410060280 2024-01-21T08:08:31.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.144+0000 7f3414e49700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f33fc02ff60 0x7f33fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:31.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.145+0000 7f3414e49700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f33fc02ff60 0x7f33fc032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f340c0096a0 tx=0x7f340c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:31.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.149+0000 7f3405ffb700 1 -- 172.21.15.78:0/3658617505 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34000095f0 con 0x7f3410060280 2024-01-21T08:08:31.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.354+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34101080a0 con 0x7f3410060280 2024-01-21T08:08:31.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.355+0000 7f3405ffb700 1 -- 172.21.15.78:0/3658617505 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f340001cb10 con 0x7f3410060280 2024-01-21T08:08:31.357 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:31.357 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:31.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f33fc02ff60 msgr2=0x7f33fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:31.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f33fc02ff60 0x7f33fc032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f340c0096a0 tx=0x7f340c006b40).stop 2024-01-21T08:08:31.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 msgr2=0x7f34101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34101066e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f3400013510 tx=0x7f340000b040).stop 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 shutdown_connections 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f33fc02ff60 0x7f33fc032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 --2- 172.21.15.78:0/3658617505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410060280 0x7f34101066e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 >> 172.21.15.78:0/3658617505 conn(0x7f34100f0b70 msgr2=0x7f34100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 shutdown_connections 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:31.356+0000 7f34178ae700 1 -- 172.21.15.78:0/3658617505 wait complete. 2024-01-21T08:08:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:32.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:32.739 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:30.619741+0000 mgr.smithi046.lmxnwc (mgr.14164) 102 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:31.355842+0000 mon.smithi046 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.78:0/3658617505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:33.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.821+0000 7f844d5fc700 1 -- 172.21.15.78:0/3562401395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 msgr2=0x7f84480f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:33.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.821+0000 7f844d5fc700 1 --2- 172.21.15.78:0/3562401395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f84480f4be0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f843c009fb0 tx=0x7f843c009b30).stop 2024-01-21T08:08:33.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.821+0000 7f844d5fc700 1 -- 172.21.15.78:0/3562401395 shutdown_connections 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.821+0000 7f844d5fc700 1 --2- 172.21.15.78:0/3562401395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f84480f4be0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.821+0000 7f844d5fc700 1 -- 172.21.15.78:0/3562401395 >> 172.21.15.78:0/3562401395 conn(0x7f84480f02e0 msgr2=0x7f84480f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.822+0000 7f844d5fc700 1 -- 172.21.15.78:0/3562401395 shutdown_connections 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.822+0000 7f844d5fc700 1 -- 172.21.15.78:0/3562401395 wait complete. 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.822+0000 7f844d5fc700 1 Processor -- start 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.822+0000 7f844d5fc700 1 -- start start 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.822+0000 7f844d5fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f8448106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.822+0000 7f844d5fc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8448106bf0 con 0x7f84480f47c0 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f8446ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f8448106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f8446ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f8448106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:38864/0 (socket says 172.21.15.78:38864) 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f8446ffd700 1 -- 172.21.15.78:0/3813714627 learned_addr learned my addr 172.21.15.78:0/3813714627 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f8446ffd700 1 -- 172.21.15.78:0/3813714627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c005040 con 0x7f84480f47c0 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f8446ffd700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f8448106700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f843c013510 tx=0x7f843c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f842f7fe700 1 -- 172.21.15.78:0/3813714627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f843c009320 con 0x7f84480f47c0 2024-01-21T08:08:33.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f842f7fe700 1 -- 172.21.15.78:0/3813714627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f843c013f00 con 0x7f84480f47c0 2024-01-21T08:08:33.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8448106df0 con 0x7f84480f47c0 2024-01-21T08:08:33.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f842f7fe700 1 -- 172.21.15.78:0/3813714627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f843c014d90 con 0x7f84480f47c0 2024-01-21T08:08:33.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.823+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8448107280 con 0x7f84480f47c0 2024-01-21T08:08:33.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.824+0000 7f842f7fe700 1 -- 172.21.15.78:0/3813714627 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f843c014420 con 0x7f84480f47c0 2024-01-21T08:08:33.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.824+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8448045ad0 con 0x7f84480f47c0 2024-01-21T08:08:33.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.824+0000 7f842f7fe700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f843002ff60 0x7f8430032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:33.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.824+0000 7f842f7fe700 1 -- 172.21.15.78:0/3813714627 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f843c041000 con 0x7f84480f47c0 2024-01-21T08:08:33.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.825+0000 7f84467fc700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f843002ff60 0x7f8430032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:33.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.825+0000 7f84467fc700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f843002ff60 0x7f8430032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f84380096a0 tx=0x7f8438006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:33.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:33.829+0000 7f842f7fe700 1 -- 172.21.15.78:0/3813714627 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f843c0095f0 con 0x7f84480f47c0 2024-01-21T08:08:34.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.037+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8448108130 con 0x7f84480f47c0 2024-01-21T08:08:34.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.037+0000 7f842f7fe700 1 -- 172.21.15.78:0/3813714627 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f843c01db70 con 0x7f84480f47c0 2024-01-21T08:08:34.039 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:34.039 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:34.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f843002ff60 msgr2=0x7f8430032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:34.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f843002ff60 0x7f8430032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f84380096a0 tx=0x7f8438006b40).stop 2024-01-21T08:08:34.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 msgr2=0x7f8448106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:34.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f8448106700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f843c013510 tx=0x7f843c00b040).stop 2024-01-21T08:08:34.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 shutdown_connections 2024-01-21T08:08:34.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f843002ff60 0x7f8430032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:34.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 --2- 172.21.15.78:0/3813714627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84480f47c0 0x7f8448106700 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:34.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 >> 172.21.15.78:0/3813714627 conn(0x7f84480f02e0 msgr2=0x7f84480f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:34.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 shutdown_connections 2024-01-21T08:08:34.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:34.039+0000 7f844d5fc700 1 -- 172.21.15.78:0/3813714627 wait complete. 2024-01-21T08:08:34.042 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:34.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:32.620022+0000 mgr.smithi046.lmxnwc (mgr.14164) 103 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:34.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:34.038291+0000 mon.smithi046 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.78:0/3813714627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:35.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:35.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:36.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.611+0000 7f3916e39700 1 -- 172.21.15.78:0/2034439511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 msgr2=0x7f39100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:36.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.611+0000 7f3916e39700 1 --2- 172.21.15.78:0/2034439511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f39100f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f3904009fb0 tx=0x7f3904009b30).stop 2024-01-21T08:08:36.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.611+0000 7f3916e39700 1 -- 172.21.15.78:0/2034439511 shutdown_connections 2024-01-21T08:08:36.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.611+0000 7f3916e39700 1 --2- 172.21.15.78:0/2034439511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f39100f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:36.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.611+0000 7f3916e39700 1 -- 172.21.15.78:0/2034439511 >> 172.21.15.78:0/2034439511 conn(0x7f39100f1c70 msgr2=0x7f39100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:36.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.611+0000 7f3916e39700 1 -- 172.21.15.78:0/2034439511 shutdown_connections 2024-01-21T08:08:36.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.611+0000 7f3916e39700 1 -- 172.21.15.78:0/2034439511 wait complete. 2024-01-21T08:08:36.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.612+0000 7f3916e39700 1 Processor -- start 2024-01-21T08:08:36.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.612+0000 7f3916e39700 1 -- start start 2024-01-21T08:08:36.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.612+0000 7f3916e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f391006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.612+0000 7f3916e39700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3910069e20 con 0x7f39100f6510 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f3914bd5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f391006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f3914bd5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f391006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:38876/0 (socket says 172.21.15.78:38876) 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f3914bd5700 1 -- 172.21.15.78:0/4141733341 learned_addr learned my addr 172.21.15.78:0/4141733341 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f3914bd5700 1 -- 172.21.15.78:0/4141733341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904005040 con 0x7f39100f6510 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f3914bd5700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f391006b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3904013510 tx=0x7f390400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f390d7fa700 1 -- 172.21.15.78:0/4141733341 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3904009320 con 0x7f39100f6510 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3910069fc0 con 0x7f39100f6510 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f390d7fa700 1 -- 172.21.15.78:0/4141733341 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3904013f00 con 0x7f39100f6510 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f390d7fa700 1 -- 172.21.15.78:0/4141733341 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3904014d90 con 0x7f39100f6510 2024-01-21T08:08:36.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.613+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391006a450 con 0x7f39100f6510 2024-01-21T08:08:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.614+0000 7f390d7fa700 1 -- 172.21.15.78:0/4141733341 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39040263a0 con 0x7f39100f6510 2024-01-21T08:08:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.614+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3910059100 con 0x7f39100f6510 2024-01-21T08:08:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.614+0000 7f390d7fa700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f390002ff60 0x7f3900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.614+0000 7f390d7fa700 1 -- 172.21.15.78:0/4141733341 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39040413b0 con 0x7f39100f6510 2024-01-21T08:08:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.614+0000 7f390ffff700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f390002ff60 0x7f3900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.615+0000 7f390ffff700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f390002ff60 0x7f3900032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f38fc0096a0 tx=0x7f38fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.619+0000 7f390d7fa700 1 -- 172.21.15.78:0/4141733341 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3904012330 con 0x7f39100f6510 2024-01-21T08:08:36.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:34.620369+0000 mgr.smithi046.lmxnwc (mgr.14164) 104 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:36.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.825+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39100fbf60 con 0x7f39100f6510 2024-01-21T08:08:36.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.825+0000 7f390d7fa700 1 -- 172.21.15.78:0/4141733341 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3904012330 con 0x7f39100f6510 2024-01-21T08:08:36.827 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f390002ff60 msgr2=0x7f3900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f390002ff60 0x7f3900032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f38fc0096a0 tx=0x7f38fc006b40).stop 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 msgr2=0x7f391006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f391006b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3904013510 tx=0x7f390400b040).stop 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 shutdown_connections 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f390002ff60 0x7f3900032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 --2- 172.21.15.78:0/4141733341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39100f6510 0x7f391006b9a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 >> 172.21.15.78:0/4141733341 conn(0x7f39100f1c70 msgr2=0x7f39100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 shutdown_connections 2024-01-21T08:08:36.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:36.827+0000 7f3916e39700 1 -- 172.21.15.78:0/4141733341 wait complete. 2024-01-21T08:08:36.830 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:36.826454+0000 mon.smithi046 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.78:0/4141733341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:38.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:38.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:36.620658+0000 mgr.smithi046.lmxnwc (mgr.14164) 105 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 -- 172.21.15.78:0/2570698421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 msgr2=0x7fb6040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 --2- 172.21.15.78:0/2570698421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb6040f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb5f8009fb0 tx=0x7fb5f8009b30).stop 2024-01-21T08:08:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 -- 172.21.15.78:0/2570698421 shutdown_connections 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 --2- 172.21.15.78:0/2570698421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb6040f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 -- 172.21.15.78:0/2570698421 >> 172.21.15.78:0/2570698421 conn(0x7fb6040f0b70 msgr2=0x7fb6040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 -- 172.21.15.78:0/2570698421 shutdown_connections 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 -- 172.21.15.78:0/2570698421 wait complete. 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 Processor -- start 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.346+0000 7fb60aeac700 1 -- start start 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.347+0000 7fb60aeac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb604106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.347+0000 7fb60aeac700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604106c10 con 0x7fb6040f5410 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.347+0000 7fb608c48700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb604106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.347+0000 7fb608c48700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb604106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:38886/0 (socket says 172.21.15.78:38886) 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.347+0000 7fb608c48700 1 -- 172.21.15.78:0/2383524205 learned_addr learned my addr 172.21.15.78:0/2383524205 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.347+0000 7fb608c48700 1 -- 172.21.15.78:0/2383524205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f8005040 con 0x7fb6040f5410 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.347+0000 7fb608c48700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb604106720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb5f8014000 tx=0x7fb5f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.348+0000 7fb6017fa700 1 -- 172.21.15.78:0/2383524205 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5f80095f0 con 0x7fb6040f5410 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.348+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb604106e10 con 0x7fb6040f5410 2024-01-21T08:08:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.348+0000 7fb6017fa700 1 -- 172.21.15.78:0/2383524205 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5f8007e70 con 0x7fb6040f5410 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.348+0000 7fb6017fa700 1 -- 172.21.15.78:0/2383524205 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5f8014ea0 con 0x7fb6040f5410 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.348+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6041072a0 con 0x7fb6040f5410 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.348+0000 7fb6017fa700 1 -- 172.21.15.78:0/2383524205 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5f8025070 con 0x7fb6040f5410 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.349+0000 7fb6017fa700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb5f402ff60 0x7fb5f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.349+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb604045ad0 con 0x7fb6040f5410 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.349+0000 7fb6017fa700 1 -- 172.21.15.78:0/2383524205 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5f8040930 con 0x7fb6040f5410 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.349+0000 7fb603fff700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb5f402ff60 0x7fb5f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.349+0000 7fb603fff700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb5f402ff60 0x7fb5f4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb5f00096a0 tx=0x7fb5f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.354+0000 7fb6017fa700 1 -- 172.21.15.78:0/2383524205 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5f800f4e0 con 0x7fb6040f5410 2024-01-21T08:08:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.560+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb604108150 con 0x7fb6040f5410 2024-01-21T08:08:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.561+0000 7fb6017fa700 1 -- 172.21.15.78:0/2383524205 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb5f800f300 con 0x7fb6040f5410 2024-01-21T08:08:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:39.564 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.562+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb5f402ff60 msgr2=0x7fb5f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.562+0000 7fb60aeac700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb5f402ff60 0x7fb5f4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb5f00096a0 tx=0x7fb5f0006b40).stop 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.562+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 msgr2=0x7fb604106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.562+0000 7fb60aeac700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb604106720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb5f8014000 tx=0x7fb5f800b040).stop 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.563+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 shutdown_connections 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.563+0000 7fb60aeac700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb5f402ff60 0x7fb5f4032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.563+0000 7fb60aeac700 1 --2- 172.21.15.78:0/2383524205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6040f5410 0x7fb604106720 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.563+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 >> 172.21.15.78:0/2383524205 conn(0x7fb6040f0b70 msgr2=0x7fb60405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.563+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 shutdown_connections 2024-01-21T08:08:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:39.563+0000 7fb60aeac700 1 -- 172.21.15.78:0/2383524205 wait complete. 2024-01-21T08:08:39.567 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:38.620907+0000 mgr.smithi046.lmxnwc (mgr.14164) 106 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:40.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:39.563066+0000 mon.smithi046 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.78:0/2383524205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:40.951 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:40.952 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:42.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.094+0000 7f103c8cb700 1 -- 172.21.15.78:0/1382593363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 msgr2=0x7f10380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:42.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.094+0000 7f103c8cb700 1 --2- 172.21.15.78:0/1382593363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10380f55c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f1028009fb0 tx=0x7f1028009b30).stop 2024-01-21T08:08:42.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.095+0000 7f103c8cb700 1 -- 172.21.15.78:0/1382593363 shutdown_connections 2024-01-21T08:08:42.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.095+0000 7f103c8cb700 1 --2- 172.21.15.78:0/1382593363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10380f55c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:42.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.095+0000 7f103c8cb700 1 -- 172.21.15.78:0/1382593363 >> 172.21.15.78:0/1382593363 conn(0x7f10380f09a0 msgr2=0x7f10380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:42.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.095+0000 7f103c8cb700 1 -- 172.21.15.78:0/1382593363 shutdown_connections 2024-01-21T08:08:42.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.095+0000 7f103c8cb700 1 -- 172.21.15.78:0/1382593363 wait complete. 2024-01-21T08:08:42.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.095+0000 7f103c8cb700 1 Processor -- start 2024-01-21T08:08:42.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.095+0000 7f103c8cb700 1 -- start start 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.096+0000 7f103c8cb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10381091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.096+0000 7f103c8cb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1038060240 con 0x7f10380f51a0 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.096+0000 7f10377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10381091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.096+0000 7f10377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10381091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:38906/0 (socket says 172.21.15.78:38906) 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.096+0000 7f10377fe700 1 -- 172.21.15.78:0/4253515705 learned_addr learned my addr 172.21.15.78:0/4253515705 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.096+0000 7f10377fe700 1 -- 172.21.15.78:0/4253515705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028005040 con 0x7f10380f51a0 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.097+0000 7f10377fe700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10381091b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1028013510 tx=0x7f102800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.097+0000 7f101ffff700 1 -- 172.21.15.78:0/4253515705 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1028009320 con 0x7f10380f51a0 2024-01-21T08:08:42.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.097+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1038060440 con 0x7f10380f51a0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.097+0000 7f101ffff700 1 -- 172.21.15.78:0/4253515705 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1028013f00 con 0x7f10380f51a0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.097+0000 7f101ffff700 1 -- 172.21.15.78:0/4253515705 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1028014dc0 con 0x7f10380f51a0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.097+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1038109860 con 0x7f10380f51a0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.097+0000 7f101ffff700 1 -- 172.21.15.78:0/4253515705 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f102801d3d0 con 0x7f10380f51a0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.098+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1038045ad0 con 0x7f10380f51a0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.098+0000 7f101ffff700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f102002ffb0 0x7f1020032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.098+0000 7f101ffff700 1 -- 172.21.15.78:0/4253515705 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10280413e0 con 0x7f10380f51a0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.098+0000 7f1036ffd700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f102002ffb0 0x7f1020032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:42.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.098+0000 7f1036ffd700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f102002ffb0 0x7f1020032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f102c0096a0 tx=0x7f102c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:42.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.103+0000 7f101ffff700 1 -- 172.21.15.78:0/4253515705 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f102801cb10 con 0x7f10380f51a0 2024-01-21T08:08:42.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.307+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f103810a080 con 0x7f10380f51a0 2024-01-21T08:08:42.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.308+0000 7f101ffff700 1 -- 172.21.15.78:0/4253515705 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f102801c930 con 0x7f10380f51a0 2024-01-21T08:08:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:42.311 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:42.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f102002ffb0 msgr2=0x7f1020032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:42.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f102002ffb0 0x7f1020032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f102c0096a0 tx=0x7f102c006b40).stop 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 msgr2=0x7f10381091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10381091b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1028013510 tx=0x7f102800b040).stop 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 shutdown_connections 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f102002ffb0 0x7f1020032470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 --2- 172.21.15.78:0/4253515705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10380f51a0 0x7f10381091b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 >> 172.21.15.78:0/4253515705 conn(0x7f10380f09a0 msgr2=0x7f10380f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 shutdown_connections 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:42.309+0000 7f103c8cb700 1 -- 172.21.15.78:0/4253515705 wait complete. 2024-01-21T08:08:42.313 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:40.621263+0000 mgr.smithi046.lmxnwc (mgr.14164) 107 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:42.309934+0000 mon.smithi046 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.78:0/4253515705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:43.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:43.704 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:42.621700+0000 mgr.smithi046.lmxnwc (mgr.14164) 108 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.910+0000 7f0e816ca700 1 -- 172.21.15.78:0/2449793104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 msgr2=0x7f0e7c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.910+0000 7f0e816ca700 1 --2- 172.21.15.78:0/2449793104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f4be0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f0e70009fb0 tx=0x7f0e70009b30).stop 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.910+0000 7f0e816ca700 1 -- 172.21.15.78:0/2449793104 shutdown_connections 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.910+0000 7f0e816ca700 1 --2- 172.21.15.78:0/2449793104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f4be0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.910+0000 7f0e816ca700 1 -- 172.21.15.78:0/2449793104 >> 172.21.15.78:0/2449793104 conn(0x7f0e7c0f02e0 msgr2=0x7f0e7c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.910+0000 7f0e816ca700 1 -- 172.21.15.78:0/2449793104 shutdown_connections 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.910+0000 7f0e816ca700 1 -- 172.21.15.78:0/2449793104 wait complete. 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.911+0000 7f0e816ca700 1 Processor -- start 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.911+0000 7f0e816ca700 1 -- start start 2024-01-21T08:08:44.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.911+0000 7f0e816ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.911+0000 7f0e816ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0f57f0 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.911+0000 7f0e7affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.911+0000 7f0e7affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:58096/0 (socket says 172.21.15.78:58096) 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.911+0000 7f0e7affd700 1 -- 172.21.15.78:0/2845438835 learned_addr learned my addr 172.21.15.78:0/2845438835 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.912+0000 7f0e7affd700 1 -- 172.21.15.78:0/2845438835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70005040 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.912+0000 7f0e7affd700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f7370 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f0e70013510 tx=0x7f0e7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.912+0000 7f0e637fe700 1 -- 172.21.15.78:0/2845438835 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e70009320 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.912+0000 7f0e637fe700 1 -- 172.21.15.78:0/2845438835 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e70013f00 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.912+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7c0f5990 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.912+0000 7f0e637fe700 1 -- 172.21.15.78:0/2845438835 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e70014d90 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.912+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7c0f5e20 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.913+0000 7f0e637fe700 1 -- 172.21.15.78:0/2845438835 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e700263a0 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.913+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7c059100 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.913+0000 7f0e637fe700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0e6402ff60 0x7f0e64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:44.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.913+0000 7f0e637fe700 1 -- 172.21.15.78:0/2845438835 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e70041010 con 0x7f0e7c0f47c0 2024-01-21T08:08:44.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.913+0000 7f0e7a7fc700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0e6402ff60 0x7f0e64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:44.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.914+0000 7f0e7a7fc700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0e6402ff60 0x7f0e64032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0e6c0096a0 tx=0x7f0e6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:44.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:44.918+0000 7f0e637fe700 1 -- 172.21.15.78:0/2845438835 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e700126f0 con 0x7f0e7c0f47c0 2024-01-21T08:08:45.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.121+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e7c0fbf60 con 0x7f0e7c0f47c0 2024-01-21T08:08:45.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.121+0000 7f0e637fe700 1 -- 172.21.15.78:0/2845438835 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e70012330 con 0x7f0e7c0f47c0 2024-01-21T08:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0e6402ff60 msgr2=0x7f0e64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0e6402ff60 0x7f0e64032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0e6c0096a0 tx=0x7f0e6c006b40).stop 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 msgr2=0x7f0e7c0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f7370 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f0e70013510 tx=0x7f0e7000b040).stop 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 shutdown_connections 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0e6402ff60 0x7f0e64032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 --2- 172.21.15.78:0/2845438835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e7c0f47c0 0x7f0e7c0f7370 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 >> 172.21.15.78:0/2845438835 conn(0x7f0e7c0f02e0 msgr2=0x7f0e7c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:45.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 shutdown_connections 2024-01-21T08:08:45.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:45.123+0000 7f0e816ca700 1 -- 172.21.15.78:0/2845438835 wait complete. 2024-01-21T08:08:45.127 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:45.122308+0000 mon.smithi046 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.78:0/2845438835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:46.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:46.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:46.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:44.622068+0000 mgr.smithi046.lmxnwc (mgr.14164) 109 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.723+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3074415427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 msgr2=0x7fc1a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.723+0000 7fc1adbfb700 1 --2- 172.21.15.78:0/3074415427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f90f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fc198009fb0 tx=0x7fc198009b30).stop 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.723+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3074415427 shutdown_connections 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.723+0000 7fc1adbfb700 1 --2- 172.21.15.78:0/3074415427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f90f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.723+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3074415427 >> 172.21.15.78:0/3074415427 conn(0x7fc1a80f0b70 msgr2=0x7fc1a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.723+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3074415427 shutdown_connections 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.723+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3074415427 wait complete. 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.724+0000 7fc1adbfb700 1 Processor -- start 2024-01-21T08:08:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.724+0000 7fc1adbfb700 1 -- start start 2024-01-21T08:08:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.724+0000 7fc1adbfb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.724+0000 7fc1adbfb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80f4ff0 con 0x7fc1a8060280 2024-01-21T08:08:47.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.724+0000 7fc1a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:47.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.724+0000 7fc1a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:58112/0 (socket says 172.21.15.78:58112) 2024-01-21T08:08:47.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.724+0000 7fc1a77fe700 1 -- 172.21.15.78:0/3653387555 learned_addr learned my addr 172.21.15.78:0/3653387555 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.725+0000 7fc1a77fe700 1 -- 172.21.15.78:0/3653387555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198005040 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.725+0000 7fc1a77fe700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f6b70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fc198013510 tx=0x7fc19800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.725+0000 7fc19ffff700 1 -- 172.21.15.78:0/3653387555 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc198009320 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.725+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a80f5190 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.725+0000 7fc19ffff700 1 -- 172.21.15.78:0/3653387555 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc198013f00 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.725+0000 7fc19ffff700 1 -- 172.21.15.78:0/3653387555 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc198014dc0 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.725+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a80f5620 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.726+0000 7fc19ffff700 1 -- 172.21.15.78:0/3653387555 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1980095f0 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.726+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a8059100 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.726+0000 7fc19ffff700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc19402ff60 0x7fc194032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.726+0000 7fc19ffff700 1 -- 172.21.15.78:0/3653387555 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1980410c0 con 0x7fc1a8060280 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.726+0000 7fc1a6ffd700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc19402ff60 0x7fc194032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:47.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.727+0000 7fc1a6ffd700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc19402ff60 0x7fc194032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc1900096a0 tx=0x7fc190006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:47.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.731+0000 7fc19ffff700 1 -- 172.21.15.78:0/3653387555 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc19801c790 con 0x7fc1a8060280 2024-01-21T08:08:47.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.937+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1a8045ad0 con 0x7fc1a8060280 2024-01-21T08:08:47.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.938+0000 7fc19ffff700 1 -- 172.21.15.78:0/3653387555 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc19801c790 con 0x7fc1a8060280 2024-01-21T08:08:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:47.940 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc19402ff60 msgr2=0x7fc194032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc19402ff60 0x7fc194032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc1900096a0 tx=0x7fc190006b40).stop 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 msgr2=0x7fc1a80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f6b70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fc198013510 tx=0x7fc19800b040).stop 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 shutdown_connections 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc19402ff60 0x7fc194032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 --2- 172.21.15.78:0/3653387555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a8060280 0x7fc1a80f6b70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 >> 172.21.15.78:0/3653387555 conn(0x7fc1a80f0b70 msgr2=0x7fc1a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 shutdown_connections 2024-01-21T08:08:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:47.940+0000 7fc1adbfb700 1 -- 172.21.15.78:0/3653387555 wait complete. 2024-01-21T08:08:47.943 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:46.622433+0000 mgr.smithi046.lmxnwc (mgr.14164) 110 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:47.939030+0000 mon.smithi046 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.78:0/3653387555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:49.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:49.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.472+0000 7fe66102f700 1 -- 172.21.15.78:0/2304472672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 msgr2=0x7fe65c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.472+0000 7fe66102f700 1 --2- 172.21.15.78:0/2304472672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c0f6930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fe650009fb0 tx=0x7fe650009b30).stop 2024-01-21T08:08:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.472+0000 7fe66102f700 1 -- 172.21.15.78:0/2304472672 shutdown_connections 2024-01-21T08:08:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.472+0000 7fe66102f700 1 --2- 172.21.15.78:0/2304472672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c0f6930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.472+0000 7fe66102f700 1 -- 172.21.15.78:0/2304472672 >> 172.21.15.78:0/2304472672 conn(0x7fe65c0f1c70 msgr2=0x7fe65c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.472+0000 7fe66102f700 1 -- 172.21.15.78:0/2304472672 shutdown_connections 2024-01-21T08:08:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.472+0000 7fe66102f700 1 -- 172.21.15.78:0/2304472672 wait complete. 2024-01-21T08:08:50.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.473+0000 7fe66102f700 1 Processor -- start 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.473+0000 7fe66102f700 1 -- start start 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.473+0000 7fe66102f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.473+0000 7fe66102f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c106c20 con 0x7fe65c0f6510 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.473+0000 7fe65ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.473+0000 7fe65ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:58130/0 (socket says 172.21.15.78:58130) 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.473+0000 7fe65ad9d700 1 -- 172.21.15.78:0/1179812828 learned_addr learned my addr 172.21.15.78:0/1179812828 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.474+0000 7fe65ad9d700 1 -- 172.21.15.78:0/1179812828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe650005040 con 0x7fe65c0f6510 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.474+0000 7fe65ad9d700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fe650014000 tx=0x7fe65000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.474+0000 7fe6437fe700 1 -- 172.21.15.78:0/1179812828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6500095f0 con 0x7fe65c0f6510 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.474+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c106e20 con 0x7fe65c0f6510 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.474+0000 7fe6437fe700 1 -- 172.21.15.78:0/1179812828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe650007e70 con 0x7fe65c0f6510 2024-01-21T08:08:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.474+0000 7fe6437fe700 1 -- 172.21.15.78:0/1179812828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe650014ea0 con 0x7fe65c0f6510 2024-01-21T08:08:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.474+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c1072b0 con 0x7fe65c0f6510 2024-01-21T08:08:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.475+0000 7fe6437fe700 1 -- 172.21.15.78:0/1179812828 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe650025070 con 0x7fe65c0f6510 2024-01-21T08:08:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.475+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c045ad0 con 0x7fe65c0f6510 2024-01-21T08:08:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.475+0000 7fe6437fe700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe64402ff60 0x7fe644032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.475+0000 7fe6437fe700 1 -- 172.21.15.78:0/1179812828 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6500409c0 con 0x7fe65c0f6510 2024-01-21T08:08:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.475+0000 7fe65a59c700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe64402ff60 0x7fe644032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.475+0000 7fe65a59c700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe64402ff60 0x7fe644032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe64c0096a0 tx=0x7fe64c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.480+0000 7fe6437fe700 1 -- 172.21.15.78:0/1179812828 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe650026350 con 0x7fe65c0f6510 2024-01-21T08:08:50.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.689+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe65c0fbf60 con 0x7fe65c0f6510 2024-01-21T08:08:50.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.689+0000 7fe6437fe700 1 -- 172.21.15.78:0/1179812828 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe650012330 con 0x7fe65c0f6510 2024-01-21T08:08:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:50.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe64402ff60 msgr2=0x7fe644032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:50.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe64402ff60 0x7fe644032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe64c0096a0 tx=0x7fe64c006b40).stop 2024-01-21T08:08:50.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 msgr2=0x7fe65c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:50.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fe650014000 tx=0x7fe65000b040).stop 2024-01-21T08:08:50.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 shutdown_connections 2024-01-21T08:08:50.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe64402ff60 0x7fe644032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:50.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 --2- 172.21.15.78:0/1179812828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe65c0f6510 0x7fe65c106730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:50.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 >> 172.21.15.78:0/1179812828 conn(0x7fe65c0f1c70 msgr2=0x7fe65c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:50.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 shutdown_connections 2024-01-21T08:08:50.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:50.691+0000 7fe66102f700 1 -- 172.21.15.78:0/1179812828 wait complete. 2024-01-21T08:08:50.694 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:48.622843+0000 mgr.smithi046.lmxnwc (mgr.14164) 111 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:50.690299+0000 mon.smithi046 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.78:0/1179812828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:52.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:52.117 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:52.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:50.623192+0000 mgr.smithi046.lmxnwc (mgr.14164) 112 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.384+0000 7fdfaffff700 1 -- 172.21.15.78:0/743864773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 msgr2=0x7fdfb00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.384+0000 7fdfaffff700 1 --2- 172.21.15.78:0/743864773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb00f66c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fdfa0009fb0 tx=0x7fdfa0009b30).stop 2024-01-21T08:08:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.384+0000 7fdfaffff700 1 -- 172.21.15.78:0/743864773 shutdown_connections 2024-01-21T08:08:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.384+0000 7fdfaffff700 1 --2- 172.21.15.78:0/743864773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb00f66c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.384+0000 7fdfaffff700 1 -- 172.21.15.78:0/743864773 >> 172.21.15.78:0/743864773 conn(0x7fdfb00f1aa0 msgr2=0x7fdfb00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.385+0000 7fdfaffff700 1 -- 172.21.15.78:0/743864773 shutdown_connections 2024-01-21T08:08:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.385+0000 7fdfaffff700 1 -- 172.21.15.78:0/743864773 wait complete. 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.385+0000 7fdfaffff700 1 Processor -- start 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.385+0000 7fdfaffff700 1 -- start start 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaffff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb01069a0 con 0x7fdfb00f62a0 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:48552/0 (socket says 172.21.15.78:48552) 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaeffd700 1 -- 172.21.15.78:0/1538875910 learned_addr learned my addr 172.21.15.78:0/1538875910 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaeffd700 1 -- 172.21.15.78:0/1538875910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa0005040 con 0x7fdfb00f62a0 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaeffd700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb01064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fdfa0013510 tx=0x7fdfa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdf977fe700 1 -- 172.21.15.78:0/1538875910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa0009320 con 0x7fdfb00f62a0 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdf977fe700 1 -- 172.21.15.78:0/1538875910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfa0013f00 con 0x7fdfb00f62a0 2024-01-21T08:08:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.386+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb0106ba0 con 0x7fdfb00f62a0 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.387+0000 7fdf977fe700 1 -- 172.21.15.78:0/1538875910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa0014dc0 con 0x7fdfb00f62a0 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.387+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb0107030 con 0x7fdfb00f62a0 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.387+0000 7fdf977fe700 1 -- 172.21.15.78:0/1538875910 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfa001d3d0 con 0x7fdfb00f62a0 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.387+0000 7fdf977fe700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf9802ff60 0x7fdf98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.387+0000 7fdf977fe700 1 -- 172.21.15.78:0/1538875910 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfa00410b0 con 0x7fdfb00f62a0 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.388+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb0045ad0 con 0x7fdfb00f62a0 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.388+0000 7fdfae7fc700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf9802ff60 0x7fdf98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.388+0000 7fdfae7fc700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf9802ff60 0x7fdf98032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdfa40096a0 tx=0x7fdfa4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.392+0000 7fdf977fe700 1 -- 172.21.15.78:0/1538875910 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfa0012920 con 0x7fdfb00f62a0 2024-01-21T08:08:53.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.597+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfb0107d50 con 0x7fdfb00f62a0 2024-01-21T08:08:53.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.597+0000 7fdf977fe700 1 -- 172.21.15.78:0/1538875910 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfa0012330 con 0x7fdfb00f62a0 2024-01-21T08:08:53.599 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:53.599 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf9802ff60 msgr2=0x7fdf98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf9802ff60 0x7fdf98032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdfa40096a0 tx=0x7fdfa4006b40).stop 2024-01-21T08:08:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 msgr2=0x7fdfb01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb01064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fdfa0013510 tx=0x7fdfa000b040).stop 2024-01-21T08:08:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 shutdown_connections 2024-01-21T08:08:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf9802ff60 0x7fdf98032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 --2- 172.21.15.78:0/1538875910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfb00f62a0 0x7fdfb01064b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 >> 172.21.15.78:0/1538875910 conn(0x7fdfb00f1aa0 msgr2=0x7fdfb00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 shutdown_connections 2024-01-21T08:08:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:53.599+0000 7fdfaffff700 1 -- 172.21.15.78:0/1538875910 wait complete. 2024-01-21T08:08:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:52.623630+0000 mgr.smithi046.lmxnwc (mgr.14164) 113 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:53.598482+0000 mon.smithi046 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.78:0/1538875910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:54.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:54.991 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:56.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.118+0000 7fb9c4810700 1 -- 172.21.15.78:0/2860488949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 msgr2=0x7fb9c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:56.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.118+0000 7fb9c4810700 1 --2- 172.21.15.78:0/2860488949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c00f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fb9a8009fb0 tx=0x7fb9a8009b30).stop 2024-01-21T08:08:56.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 -- 172.21.15.78:0/2860488949 shutdown_connections 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 --2- 172.21.15.78:0/2860488949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c00f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 -- 172.21.15.78:0/2860488949 >> 172.21.15.78:0/2860488949 conn(0x7fb9c00f0b70 msgr2=0x7fb9c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 -- 172.21.15.78:0/2860488949 shutdown_connections 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 -- 172.21.15.78:0/2860488949 wait complete. 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 Processor -- start 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 -- start start 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.119+0000 7fb9c4810700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0060240 con 0x7fb9c00f5410 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9be59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9be59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:48578/0 (socket says 172.21.15.78:48578) 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9be59c700 1 -- 172.21.15.78:0/1455265935 learned_addr learned my addr 172.21.15.78:0/1455265935 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9be59c700 1 -- 172.21.15.78:0/1455265935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a8005040 con 0x7fb9c00f5410 2024-01-21T08:08:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9be59c700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c0109430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb9a8013510 tx=0x7fb9a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:56.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9b6ffd700 1 -- 172.21.15.78:0/1455265935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a8009320 con 0x7fb9c00f5410 2024-01-21T08:08:56.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c0060440 con 0x7fb9c00f5410 2024-01-21T08:08:56.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9b6ffd700 1 -- 172.21.15.78:0/1455265935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9a8013f00 con 0x7fb9c00f5410 2024-01-21T08:08:56.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9b6ffd700 1 -- 172.21.15.78:0/1455265935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a8014d90 con 0x7fb9c00f5410 2024-01-21T08:08:56.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.120+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c0109ae0 con 0x7fb9c00f5410 2024-01-21T08:08:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.121+0000 7fb9b6ffd700 1 -- 172.21.15.78:0/1455265935 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9a80095f0 con 0x7fb9c00f5410 2024-01-21T08:08:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.121+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c0045ad0 con 0x7fb9c00f5410 2024-01-21T08:08:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.121+0000 7fb9b6ffd700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb9ac02ff60 0x7fb9ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.121+0000 7fb9b6ffd700 1 -- 172.21.15.78:0/1455265935 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9a80410c0 con 0x7fb9c00f5410 2024-01-21T08:08:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.121+0000 7fb9bdd9b700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb9ac02ff60 0x7fb9ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.122+0000 7fb9bdd9b700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb9ac02ff60 0x7fb9ac032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb9b00096a0 tx=0x7fb9b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.126+0000 7fb9b6ffd700 1 -- 172.21.15.78:0/1455265935 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9a8014420 con 0x7fb9c00f5410 2024-01-21T08:08:56.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.336+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9c00fae60 con 0x7fb9c00f5410 2024-01-21T08:08:56.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.337+0000 7fb9b6ffd700 1 -- 172.21.15.78:0/1455265935 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9a801d7f0 con 0x7fb9c00f5410 2024-01-21T08:08:56.339 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:56.339 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb9ac02ff60 msgr2=0x7fb9ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb9ac02ff60 0x7fb9ac032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb9b00096a0 tx=0x7fb9b0006b40).stop 2024-01-21T08:08:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 msgr2=0x7fb9c0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c0109430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb9a8013510 tx=0x7fb9a800b040).stop 2024-01-21T08:08:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 shutdown_connections 2024-01-21T08:08:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb9ac02ff60 0x7fb9ac032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 --2- 172.21.15.78:0/1455265935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00f5410 0x7fb9c0109430 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 >> 172.21.15.78:0/1455265935 conn(0x7fb9c00f0b70 msgr2=0x7fb9c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 shutdown_connections 2024-01-21T08:08:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:56.338+0000 7fb9c4810700 1 -- 172.21.15.78:0/1455265935 wait complete. 2024-01-21T08:08:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:54.623980+0000 mgr.smithi046.lmxnwc (mgr.14164) 114 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:56.337777+0000 mon.smithi046 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.78:0/1455265935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:08:57.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:08:57.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:08:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:56.624357+0000 mgr.smithi046.lmxnwc (mgr.14164) 115 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:08:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.944+0000 7f2a1a513700 1 -- 172.21.15.78:0/3314865363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 msgr2=0x7f2a140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.944+0000 7f2a1a513700 1 --2- 172.21.15.78:0/3314865363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a140f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f2a08009fb0 tx=0x7f2a08009b30).stop 2024-01-21T08:08:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.944+0000 7f2a1a513700 1 -- 172.21.15.78:0/3314865363 shutdown_connections 2024-01-21T08:08:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.944+0000 7f2a1a513700 1 --2- 172.21.15.78:0/3314865363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a140f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.944+0000 7f2a1a513700 1 -- 172.21.15.78:0/3314865363 >> 172.21.15.78:0/3314865363 conn(0x7f2a140f1c70 msgr2=0x7f2a140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.944+0000 7f2a1a513700 1 -- 172.21.15.78:0/3314865363 shutdown_connections 2024-01-21T08:08:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.944+0000 7f2a1a513700 1 -- 172.21.15.78:0/3314865363 wait complete. 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a1a513700 1 Processor -- start 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a1a513700 1 -- start start 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a1a513700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a1406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a1a513700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a14069e20 con 0x7f2a140f6510 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a1406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a1406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:48600/0 (socket says 172.21.15.78:48600) 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a13fff700 1 -- 172.21.15.78:0/2381857528 learned_addr learned my addr 172.21.15.78:0/2381857528 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.945+0000 7f2a13fff700 1 -- 172.21.15.78:0/2381857528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08005040 con 0x7f2a140f6510 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.946+0000 7f2a13fff700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a1406b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2a08013510 tx=0x7f2a0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.946+0000 7f2a10ff9700 1 -- 172.21.15.78:0/2381857528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a08009320 con 0x7f2a140f6510 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.946+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a14069fc0 con 0x7f2a140f6510 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.946+0000 7f2a10ff9700 1 -- 172.21.15.78:0/2381857528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a08013f00 con 0x7f2a140f6510 2024-01-21T08:08:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.946+0000 7f2a10ff9700 1 -- 172.21.15.78:0/2381857528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a08014d90 con 0x7f2a140f6510 2024-01-21T08:08:58.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.946+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1406a450 con 0x7f2a140f6510 2024-01-21T08:08:58.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.947+0000 7f2a10ff9700 1 -- 172.21.15.78:0/2381857528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a08014420 con 0x7f2a140f6510 2024-01-21T08:08:58.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.947+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a14059100 con 0x7f2a140f6510 2024-01-21T08:08:58.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.947+0000 7f2a10ff9700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f29fc02ff60 0x7f29fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:08:58.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.947+0000 7f2a10ff9700 1 -- 172.21.15.78:0/2381857528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a08041000 con 0x7f2a140f6510 2024-01-21T08:08:58.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.947+0000 7f2a137fe700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f29fc02ff60 0x7f29fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:08:58.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.947+0000 7f2a137fe700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f29fc02ff60 0x7f29fc032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2a040096a0 tx=0x7f2a04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:08:58.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:58.952+0000 7f2a10ff9700 1 -- 172.21.15.78:0/2381857528 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a080095f0 con 0x7f2a140f6510 2024-01-21T08:08:59.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.155+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a140fbf60 con 0x7f2a140f6510 2024-01-21T08:08:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.156+0000 7f2a10ff9700 1 -- 172.21.15.78:0/2381857528 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a0801db70 con 0x7f2a140f6510 2024-01-21T08:08:59.158 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:08:59.158 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f29fc02ff60 msgr2=0x7f29fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f29fc02ff60 0x7f29fc032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2a040096a0 tx=0x7f2a04006b40).stop 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 msgr2=0x7f2a1406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a1406b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2a08013510 tx=0x7f2a0800b040).stop 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 shutdown_connections 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f29fc02ff60 0x7f29fc032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 --2- 172.21.15.78:0/2381857528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a140f6510 0x7f2a1406b9a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 >> 172.21.15.78:0/2381857528 conn(0x7f2a140f1c70 msgr2=0x7f2a140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:08:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 shutdown_connections 2024-01-21T08:08:59.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:08:59.158+0000 7f2a1a513700 1 -- 172.21.15.78:0/2381857528 wait complete. 2024-01-21T08:08:59.161 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:08:59.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:08:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:08:59.157146+0000 mon.smithi046 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.78:0/2381857528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:00.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:00.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:08:58.624680+0000 mgr.smithi046.lmxnwc (mgr.14164) 116 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.780+0000 7fb25c68a700 1 -- 172.21.15.78:0/604592614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 msgr2=0x7fb2540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.780+0000 7fb25c68a700 1 --2- 172.21.15.78:0/604592614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb2540f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fb248009fb0 tx=0x7fb248009b30).stop 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.780+0000 7fb25c68a700 1 -- 172.21.15.78:0/604592614 shutdown_connections 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.780+0000 7fb25c68a700 1 --2- 172.21.15.78:0/604592614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb2540f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.780+0000 7fb25c68a700 1 -- 172.21.15.78:0/604592614 >> 172.21.15.78:0/604592614 conn(0x7fb2540f1c70 msgr2=0x7fb2540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.780+0000 7fb25c68a700 1 -- 172.21.15.78:0/604592614 shutdown_connections 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.780+0000 7fb25c68a700 1 -- 172.21.15.78:0/604592614 wait complete. 2024-01-21T08:09:01.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.781+0000 7fb25c68a700 1 Processor -- start 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.781+0000 7fb25c68a700 1 -- start start 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.781+0000 7fb25c68a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb25406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.781+0000 7fb25c68a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb254069e20 con 0x7fb2540f6510 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.781+0000 7fb25a426700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb25406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.781+0000 7fb25a426700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb25406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:48618/0 (socket says 172.21.15.78:48618) 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.781+0000 7fb25a426700 1 -- 172.21.15.78:0/161787499 learned_addr learned my addr 172.21.15.78:0/161787499 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.782+0000 7fb25a426700 1 -- 172.21.15.78:0/161787499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248005040 con 0x7fb2540f6510 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.782+0000 7fb25a426700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb25406b9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb248013510 tx=0x7fb24800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.782+0000 7fb246ffd700 1 -- 172.21.15.78:0/161787499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb248009320 con 0x7fb2540f6510 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.782+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb254069fc0 con 0x7fb2540f6510 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.782+0000 7fb246ffd700 1 -- 172.21.15.78:0/161787499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb248013f00 con 0x7fb2540f6510 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.782+0000 7fb246ffd700 1 -- 172.21.15.78:0/161787499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb248014d90 con 0x7fb2540f6510 2024-01-21T08:09:01.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.782+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25406a450 con 0x7fb2540f6510 2024-01-21T08:09:01.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.783+0000 7fb246ffd700 1 -- 172.21.15.78:0/161787499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2480095f0 con 0x7fb2540f6510 2024-01-21T08:09:01.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.783+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb254059100 con 0x7fb2540f6510 2024-01-21T08:09:01.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.783+0000 7fb246ffd700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb24002ff60 0x7fb240032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:01.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.783+0000 7fb246ffd700 1 -- 172.21.15.78:0/161787499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2480410c0 con 0x7fb2540f6510 2024-01-21T08:09:01.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.783+0000 7fb259c25700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb24002ff60 0x7fb240032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:01.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.783+0000 7fb259c25700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb24002ff60 0x7fb240032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb2500096a0 tx=0x7fb250006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:01.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.788+0000 7fb246ffd700 1 -- 172.21.15.78:0/161787499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb248014420 con 0x7fb2540f6510 2024-01-21T08:09:01.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.991+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2540fbf60 con 0x7fb2540f6510 2024-01-21T08:09:01.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.992+0000 7fb246ffd700 1 -- 172.21.15.78:0/161787499 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb24801d7f0 con 0x7fb2540f6510 2024-01-21T08:09:01.994 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:01.994 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:01.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb24002ff60 msgr2=0x7fb240032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:01.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb24002ff60 0x7fb240032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb2500096a0 tx=0x7fb250006b40).stop 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 msgr2=0x7fb25406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb25406b9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb248013510 tx=0x7fb24800b040).stop 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 shutdown_connections 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb24002ff60 0x7fb240032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 --2- 172.21.15.78:0/161787499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2540f6510 0x7fb25406b9a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.993+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 >> 172.21.15.78:0/161787499 conn(0x7fb2540f1c70 msgr2=0x7fb2540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.994+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 shutdown_connections 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:01.994+0000 7fb25c68a700 1 -- 172.21.15.78:0/161787499 wait complete. 2024-01-21T08:09:01.996 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:00.624973+0000 mgr.smithi046.lmxnwc (mgr.14164) 117 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:01.993032+0000 mon.smithi046 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.78:0/161787499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:03.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:03.392 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:04.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 -- 172.21.15.78:0/4082895084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 msgr2=0x7f9a180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:04.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 --2- 172.21.15.78:0/4082895084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a180f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f9a08009fb0 tx=0x7f9a08009b30).stop 2024-01-21T08:09:04.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 -- 172.21.15.78:0/4082895084 shutdown_connections 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 --2- 172.21.15.78:0/4082895084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a180f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 -- 172.21.15.78:0/4082895084 >> 172.21.15.78:0/4082895084 conn(0x7f9a180f0b70 msgr2=0x7f9a180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 -- 172.21.15.78:0/4082895084 shutdown_connections 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 -- 172.21.15.78:0/4082895084 wait complete. 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 Processor -- start 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.578+0000 7f9a1de57700 1 -- start start 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a1de57700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a18105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:04.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a1de57700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a18105af0 con 0x7f9a180f5410 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a18105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a18105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:38362/0 (socket says 172.21.15.78:38362) 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a177fe700 1 -- 172.21.15.78:0/884582802 learned_addr learned my addr 172.21.15.78:0/884582802 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a177fe700 1 -- 172.21.15.78:0/884582802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08005040 con 0x7f9a180f5410 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a177fe700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a18105600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f9a08014000 tx=0x7f9a0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f99fffff700 1 -- 172.21.15.78:0/884582802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a080095f0 con 0x7f9a180f5410 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.579+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a18105cf0 con 0x7f9a180f5410 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.580+0000 7f99fffff700 1 -- 172.21.15.78:0/884582802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a08007e70 con 0x7f9a180f5410 2024-01-21T08:09:04.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.580+0000 7f99fffff700 1 -- 172.21.15.78:0/884582802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a08014ea0 con 0x7f9a180f5410 2024-01-21T08:09:04.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.580+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a18106180 con 0x7f9a180f5410 2024-01-21T08:09:04.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.580+0000 7f99fffff700 1 -- 172.21.15.78:0/884582802 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a08025070 con 0x7f9a180f5410 2024-01-21T08:09:04.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.580+0000 7f99fffff700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a0002ffb0 0x7f9a00032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:04.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.581+0000 7f99fffff700 1 -- 172.21.15.78:0/884582802 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a08040a30 con 0x7f9a180f5410 2024-01-21T08:09:04.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.580+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a18045ad0 con 0x7f9a180f5410 2024-01-21T08:09:04.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.581+0000 7f9a16ffd700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a0002ffb0 0x7f9a00032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:04.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.581+0000 7f9a16ffd700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a0002ffb0 0x7f9a00032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f9a0c0096a0 tx=0x7f9a0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:04.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.585+0000 7f99fffff700 1 -- 172.21.15.78:0/884582802 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a08026350 con 0x7f9a180f5410 2024-01-21T08:09:04.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:02.625328+0000 mgr.smithi046.lmxnwc (mgr.14164) 118 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:04.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.790+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a180fae60 con 0x7f9a180f5410 2024-01-21T08:09:04.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.790+0000 7f99fffff700 1 -- 172.21.15.78:0/884582802 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a08012330 con 0x7f9a180f5410 2024-01-21T08:09:04.792 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:04.792 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a0002ffb0 msgr2=0x7f9a00032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a0002ffb0 0x7f9a00032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f9a0c0096a0 tx=0x7f9a0c006b40).stop 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 msgr2=0x7f9a18105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a18105600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f9a08014000 tx=0x7f9a0800b040).stop 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 shutdown_connections 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a0002ffb0 0x7f9a00032470 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 --2- 172.21.15.78:0/884582802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a180f5410 0x7f9a18105600 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 >> 172.21.15.78:0/884582802 conn(0x7f9a180f0b70 msgr2=0x7f9a180f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:04.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 shutdown_connections 2024-01-21T08:09:04.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:04.792+0000 7f9a1de57700 1 -- 172.21.15.78:0/884582802 wait complete. 2024-01-21T08:09:04.795 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:04.791430+0000 mon.smithi046 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.78:0/884582802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:06.212 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:06.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:04.625638+0000 mgr.smithi046.lmxnwc (mgr.14164) 119 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:07.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 -- 172.21.15.78:0/32643054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 msgr2=0x7f61880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 --2- 172.21.15.78:0/32643054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f90f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f6178009fb0 tx=0x7f6178009b30).stop 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 -- 172.21.15.78:0/32643054 shutdown_connections 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 --2- 172.21.15.78:0/32643054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f90f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 -- 172.21.15.78:0/32643054 >> 172.21.15.78:0/32643054 conn(0x7f61880f0b70 msgr2=0x7f61880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 -- 172.21.15.78:0/32643054 shutdown_connections 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 -- 172.21.15.78:0/32643054 wait complete. 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.409+0000 7f618e5b4700 1 Processor -- start 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f618e5b4700 1 -- start start 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f618e5b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f618e5b4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61880f4ff0 con 0x7f6188060280 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f6187fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f6187fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:38378/0 (socket says 172.21.15.78:38378) 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f6187fff700 1 -- 172.21.15.78:0/1769288646 learned_addr learned my addr 172.21.15.78:0/1769288646 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f6187fff700 1 -- 172.21.15.78:0/1769288646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6178005040 con 0x7f6188060280 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.410+0000 7f6187fff700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f6b70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6178013510 tx=0x7f617800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.411+0000 7f6184ff9700 1 -- 172.21.15.78:0/1769288646 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6178009320 con 0x7f6188060280 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.411+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61880f5190 con 0x7f6188060280 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.411+0000 7f6184ff9700 1 -- 172.21.15.78:0/1769288646 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6178013f00 con 0x7f6188060280 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.411+0000 7f6184ff9700 1 -- 172.21.15.78:0/1769288646 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6178014dc0 con 0x7f6188060280 2024-01-21T08:09:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.411+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61880f5620 con 0x7f6188060280 2024-01-21T08:09:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.412+0000 7f6184ff9700 1 -- 172.21.15.78:0/1769288646 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61780095f0 con 0x7f6188060280 2024-01-21T08:09:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.412+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6188059100 con 0x7f6188060280 2024-01-21T08:09:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.412+0000 7f6184ff9700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f617002ff60 0x7f6170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.412+0000 7f6184ff9700 1 -- 172.21.15.78:0/1769288646 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6178041000 con 0x7f6188060280 2024-01-21T08:09:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.412+0000 7f61877fe700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f617002ff60 0x7f6170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.412+0000 7f61877fe700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f617002ff60 0x7f6170032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.416+0000 7f6184ff9700 1 -- 172.21.15.78:0/1769288646 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f617801c790 con 0x7f6188060280 2024-01-21T08:09:07.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.621+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61880fbf50 con 0x7f6188060280 2024-01-21T08:09:07.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.621+0000 7f6184ff9700 1 -- 172.21.15.78:0/1769288646 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6178010230 con 0x7f6188060280 2024-01-21T08:09:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f617002ff60 msgr2=0x7f6170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f617002ff60 0x7f6170032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).stop 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 msgr2=0x7f61880f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f6b70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6178013510 tx=0x7f617800b040).stop 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 shutdown_connections 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f617002ff60 0x7f6170032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 --2- 172.21.15.78:0/1769288646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6188060280 0x7f61880f6b70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 >> 172.21.15.78:0/1769288646 conn(0x7f61880f0b70 msgr2=0x7f61880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 shutdown_connections 2024-01-21T08:09:07.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:07.623+0000 7f618e5b4700 1 -- 172.21.15.78:0/1769288646 wait complete. 2024-01-21T08:09:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:06.625991+0000 mgr.smithi046.lmxnwc (mgr.14164) 120 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:07.622423+0000 mon.smithi046 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.78:0/1769288646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:09.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:09.018 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:10.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.154+0000 7fde050ea700 1 -- 172.21.15.78:0/1779860202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 msgr2=0x7fde000f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:10.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.154+0000 7fde050ea700 1 --2- 172.21.15.78:0/1779860202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde000f5d70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fdde8009fb0 tx=0x7fdde8009b30).stop 2024-01-21T08:09:10.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.154+0000 7fde050ea700 1 -- 172.21.15.78:0/1779860202 shutdown_connections 2024-01-21T08:09:10.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.154+0000 7fde050ea700 1 --2- 172.21.15.78:0/1779860202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde000f5d70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:10.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.154+0000 7fde050ea700 1 -- 172.21.15.78:0/1779860202 >> 172.21.15.78:0/1779860202 conn(0x7fde000f0260 msgr2=0x7fde000f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:10.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.154+0000 7fde050ea700 1 -- 172.21.15.78:0/1779860202 shutdown_connections 2024-01-21T08:09:10.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.154+0000 7fde050ea700 1 -- 172.21.15.78:0/1779860202 wait complete. 2024-01-21T08:09:10.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fde050ea700 1 Processor -- start 2024-01-21T08:09:10.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fde050ea700 1 -- start start 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fde050ea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde00109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fde050ea700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde000f5310 con 0x7fde000f5950 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fddfed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde00109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fddfed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde00109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:38408/0 (socket says 172.21.15.78:38408) 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fddfed9d700 1 -- 172.21.15.78:0/1752029716 learned_addr learned my addr 172.21.15.78:0/1752029716 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.155+0000 7fddfed9d700 1 -- 172.21.15.78:0/1752029716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde8005040 con 0x7fde000f5950 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.156+0000 7fddfed9d700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde00109430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdde8013510 tx=0x7fdde800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.156+0000 7fddf77fe700 1 -- 172.21.15.78:0/1752029716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde8009320 con 0x7fde000f5950 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.156+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde000f5510 con 0x7fde000f5950 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.156+0000 7fddf77fe700 1 -- 172.21.15.78:0/1752029716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdde8013f00 con 0x7fde000f5950 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.156+0000 7fddf77fe700 1 -- 172.21.15.78:0/1752029716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde8014d90 con 0x7fde000f5950 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.156+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde00109ae0 con 0x7fde000f5950 2024-01-21T08:09:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.157+0000 7fddf77fe700 1 -- 172.21.15.78:0/1752029716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdde8014420 con 0x7fde000f5950 2024-01-21T08:09:10.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.157+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde00059100 con 0x7fde000f5950 2024-01-21T08:09:10.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.157+0000 7fddf77fe700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fddec02ff60 0x7fddec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:10.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.157+0000 7fddf77fe700 1 -- 172.21.15.78:0/1752029716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdde8041050 con 0x7fde000f5950 2024-01-21T08:09:10.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.157+0000 7fddfe59c700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fddec02ff60 0x7fddec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:10.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.158+0000 7fddfe59c700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fddec02ff60 0x7fddec032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fddf00096a0 tx=0x7fddf0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:10.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.162+0000 7fddf77fe700 1 -- 172.21.15.78:0/1752029716 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdde80095f0 con 0x7fde000f5950 2024-01-21T08:09:10.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.370+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde000fae40 con 0x7fde000f5950 2024-01-21T08:09:10.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.370+0000 7fddf77fe700 1 -- 172.21.15.78:0/1752029716 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdde801db70 con 0x7fde000f5950 2024-01-21T08:09:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:10.373 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fddec02ff60 msgr2=0x7fddec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fddec02ff60 0x7fddec032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fddf00096a0 tx=0x7fddf0006b40).stop 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 msgr2=0x7fde00109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde00109430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdde8013510 tx=0x7fdde800b040).stop 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 shutdown_connections 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fddec02ff60 0x7fddec032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 --2- 172.21.15.78:0/1752029716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde000f5950 0x7fde00109430 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 >> 172.21.15.78:0/1752029716 conn(0x7fde000f0260 msgr2=0x7fde000f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 shutdown_connections 2024-01-21T08:09:10.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:10.372+0000 7fde050ea700 1 -- 172.21.15.78:0/1752029716 wait complete. 2024-01-21T08:09:10.375 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:08.626334+0000 mgr.smithi046.lmxnwc (mgr.14164) 121 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:10.371345+0000 mon.smithi046 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.78:0/1752029716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:11.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:11.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:10.626689+0000 mgr.smithi046.lmxnwc (mgr.14164) 122 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 -- 172.21.15.78:0/3530963185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 msgr2=0x7fab9c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 --2- 172.21.15.78:0/3530963185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c0f4b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fab90009fb0 tx=0x7fab90009b30).stop 2024-01-21T08:09:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 -- 172.21.15.78:0/3530963185 shutdown_connections 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 --2- 172.21.15.78:0/3530963185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c0f4b60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 -- 172.21.15.78:0/3530963185 >> 172.21.15.78:0/3530963185 conn(0x7fab9c0f0260 msgr2=0x7fab9c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 -- 172.21.15.78:0/3530963185 shutdown_connections 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 -- 172.21.15.78:0/3530963185 wait complete. 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 Processor -- start 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.926+0000 7faba114c700 1 -- start start 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7faba114c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7faba114c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c106bc0 con 0x7fab9c0f4740 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7fab9ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7fab9ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:36668/0 (socket says 172.21.15.78:36668) 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7fab9ad9d700 1 -- 172.21.15.78:0/3183110226 learned_addr learned my addr 172.21.15.78:0/3183110226 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7fab9ad9d700 1 -- 172.21.15.78:0/3183110226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab90005040 con 0x7fab9c0f4740 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7fab9ad9d700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c1066d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fab90013510 tx=0x7fab9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.927+0000 7fab837fe700 1 -- 172.21.15.78:0/3183110226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab90009320 con 0x7fab9c0f4740 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.928+0000 7fab837fe700 1 -- 172.21.15.78:0/3183110226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab90013f00 con 0x7fab9c0f4740 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.928+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c106dc0 con 0x7fab9c0f4740 2024-01-21T08:09:12.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.928+0000 7fab837fe700 1 -- 172.21.15.78:0/3183110226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab90014d90 con 0x7fab9c0f4740 2024-01-21T08:09:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.928+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c107250 con 0x7fab9c0f4740 2024-01-21T08:09:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.928+0000 7fab837fe700 1 -- 172.21.15.78:0/3183110226 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab900095f0 con 0x7fab9c0f4740 2024-01-21T08:09:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.928+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c045ad0 con 0x7fab9c0f4740 2024-01-21T08:09:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.928+0000 7fab837fe700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab8402ff60 0x7fab84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.929+0000 7fab837fe700 1 -- 172.21.15.78:0/3183110226 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab900410c0 con 0x7fab9c0f4740 2024-01-21T08:09:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.929+0000 7fab9a59c700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab8402ff60 0x7fab84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.929+0000 7fab9a59c700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab8402ff60 0x7fab84032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:12.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:12.933+0000 7fab837fe700 1 -- 172.21.15.78:0/3183110226 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab90014420 con 0x7fab9c0f4740 2024-01-21T08:09:13.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.143+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab9c108100 con 0x7fab9c0f4740 2024-01-21T08:09:13.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.143+0000 7fab837fe700 1 -- 172.21.15.78:0/3183110226 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab9001d7f0 con 0x7fab9c0f4740 2024-01-21T08:09:13.146 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:13.146 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab8402ff60 msgr2=0x7fab84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab8402ff60 0x7fab84032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).stop 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 msgr2=0x7fab9c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c1066d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fab90013510 tx=0x7fab9000b040).stop 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 shutdown_connections 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab8402ff60 0x7fab84032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 --2- 172.21.15.78:0/3183110226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0f4740 0x7fab9c1066d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 >> 172.21.15.78:0/3183110226 conn(0x7fab9c0f0260 msgr2=0x7fab9c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 shutdown_connections 2024-01-21T08:09:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:13.145+0000 7faba114c700 1 -- 172.21.15.78:0/3183110226 wait complete. 2024-01-21T08:09:13.148 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:13.144428+0000 mon.smithi046 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.78:0/3183110226' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:14.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:14.574 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:14.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:12.626968+0000 mgr.smithi046.lmxnwc (mgr.14164) 123 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:15.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/1729472291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 msgr2=0x7fa3e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:15.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 --2- 172.21.15.78:0/1729472291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e80f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fa3dc009fb0 tx=0x7fa3dc009b30).stop 2024-01-21T08:09:15.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/1729472291 shutdown_connections 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 --2- 172.21.15.78:0/1729472291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e80f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/1729472291 >> 172.21.15.78:0/1729472291 conn(0x7fa3e80f1c70 msgr2=0x7fa3e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/1729472291 shutdown_connections 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/1729472291 wait complete. 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 Processor -- start 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.785+0000 7fa3ee8fc700 1 -- start start 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3ee8fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3ee8fc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8069e20 con 0x7fa3e80f6510 2024-01-21T08:09:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:36682/0 (socket says 172.21.15.78:36682) 2024-01-21T08:09:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3e7fff700 1 -- 172.21.15.78:0/2865510769 learned_addr learned my addr 172.21.15.78:0/2865510769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3e7fff700 1 -- 172.21.15.78:0/2865510769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc005040 con 0x7fa3e80f6510 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3e7fff700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fa3dc013510 tx=0x7fa3dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.786+0000 7fa3e4ff9700 1 -- 172.21.15.78:0/2865510769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3dc009320 con 0x7fa3e80f6510 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.787+0000 7fa3e4ff9700 1 -- 172.21.15.78:0/2865510769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3dc013f00 con 0x7fa3e80f6510 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.787+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e8069fc0 con 0x7fa3e80f6510 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.787+0000 7fa3e4ff9700 1 -- 172.21.15.78:0/2865510769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3dc014d90 con 0x7fa3e80f6510 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.787+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e806a450 con 0x7fa3e80f6510 2024-01-21T08:09:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.788+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e8059100 con 0x7fa3e80f6510 2024-01-21T08:09:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.789+0000 7fa3e4ff9700 1 -- 172.21.15.78:0/2865510769 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3dc0095f0 con 0x7fa3e80f6510 2024-01-21T08:09:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.789+0000 7fa3e4ff9700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa3d002ff60 0x7fa3d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.789+0000 7fa3e4ff9700 1 -- 172.21.15.78:0/2865510769 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3dc0411f0 con 0x7fa3e80f6510 2024-01-21T08:09:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.789+0000 7fa3e77fe700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa3d002ff60 0x7fa3d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.789+0000 7fa3e77fe700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa3d002ff60 0x7fa3d0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa3d80096a0 tx=0x7fa3d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:15.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.793+0000 7fa3e4ff9700 1 -- 172.21.15.78:0/2865510769 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3dc014420 con 0x7fa3e80f6510 2024-01-21T08:09:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.998+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3e80fbf60 con 0x7fa3e80f6510 2024-01-21T08:09:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:15.999+0000 7fa3e4ff9700 1 -- 172.21.15.78:0/2865510769 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3dc00f7b0 con 0x7fa3e80f6510 2024-01-21T08:09:16.001 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:16.001 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:16.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa3d002ff60 msgr2=0x7fa3d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:16.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa3d002ff60 0x7fa3d0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa3d80096a0 tx=0x7fa3d8006b40).stop 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 msgr2=0x7fa3e806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fa3dc013510 tx=0x7fa3dc00b040).stop 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 shutdown_connections 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa3d002ff60 0x7fa3d0032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 --2- 172.21.15.78:0/2865510769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.000+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 >> 172.21.15.78:0/2865510769 conn(0x7fa3e80f1c70 msgr2=0x7fa3e80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.001+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 shutdown_connections 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:16.001+0000 7fa3ee8fc700 1 -- 172.21.15.78:0/2865510769 wait complete. 2024-01-21T08:09:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:14.627212+0000 mgr.smithi046.lmxnwc (mgr.14164) 124 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:15.999753+0000 mon.smithi046 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.78:0/2865510769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:17.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:17.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:18.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.550+0000 7f1230a4d700 1 -- 172.21.15.78:0/1133325941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 msgr2=0x7f122c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.550+0000 7f1230a4d700 1 --2- 172.21.15.78:0/1133325941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c0f90f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f1220009fb0 tx=0x7f1220009b30).stop 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.550+0000 7f1230a4d700 1 -- 172.21.15.78:0/1133325941 shutdown_connections 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.550+0000 7f1230a4d700 1 --2- 172.21.15.78:0/1133325941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c0f90f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.550+0000 7f1230a4d700 1 -- 172.21.15.78:0/1133325941 >> 172.21.15.78:0/1133325941 conn(0x7f122c0f0b70 msgr2=0x7f122c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.550+0000 7f1230a4d700 1 -- 172.21.15.78:0/1133325941 shutdown_connections 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.550+0000 7f1230a4d700 1 -- 172.21.15.78:0/1133325941 wait complete. 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.551+0000 7f1230a4d700 1 Processor -- start 2024-01-21T08:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.551+0000 7f1230a4d700 1 -- start start 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.551+0000 7f1230a4d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.551+0000 7f1230a4d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c106bd0 con 0x7f122c060280 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.551+0000 7f122a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.551+0000 7f122a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:36694/0 (socket says 172.21.15.78:36694) 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.551+0000 7f122a59c700 1 -- 172.21.15.78:0/3247117611 learned_addr learned my addr 172.21.15.78:0/3247117611 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.552+0000 7f122a59c700 1 -- 172.21.15.78:0/3247117611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220005040 con 0x7f122c060280 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.552+0000 7f122a59c700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c1066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f1220013510 tx=0x7f122000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:18.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.552+0000 7f121affd700 1 -- 172.21.15.78:0/3247117611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1220009320 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.552+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122c106dd0 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.552+0000 7f121affd700 1 -- 172.21.15.78:0/3247117611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1220013f00 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.552+0000 7f121affd700 1 -- 172.21.15.78:0/3247117611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1220014dc0 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.552+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c107260 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.553+0000 7f121affd700 1 -- 172.21.15.78:0/3247117611 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12200095f0 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.553+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122c045ad0 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.553+0000 7f121affd700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f121402ff60 0x7f1214032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.553+0000 7f121affd700 1 -- 172.21.15.78:0/3247117611 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1220041100 con 0x7f122c060280 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.553+0000 7f1229d9b700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f121402ff60 0x7f1214032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.554+0000 7f1229d9b700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f121402ff60 0x7f1214032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f121c0096a0 tx=0x7f121c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:18.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.558+0000 7f121affd700 1 -- 172.21.15.78:0/3247117611 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1220014420 con 0x7f122c060280 2024-01-21T08:09:18.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:16.627473+0000 mgr.smithi046.lmxnwc (mgr.14164) 125 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:18.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.766+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f122c108110 con 0x7f122c060280 2024-01-21T08:09:18.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.766+0000 7f121affd700 1 -- 172.21.15.78:0/3247117611 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1220010230 con 0x7f122c060280 2024-01-21T08:09:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:18.768 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:18.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f121402ff60 msgr2=0x7f1214032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:18.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f121402ff60 0x7f1214032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f121c0096a0 tx=0x7f121c006b40).stop 2024-01-21T08:09:18.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 msgr2=0x7f122c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:18.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c1066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f1220013510 tx=0x7f122000b040).stop 2024-01-21T08:09:18.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 shutdown_connections 2024-01-21T08:09:18.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f121402ff60 0x7f1214032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:18.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 --2- 172.21.15.78:0/3247117611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f122c060280 0x7f122c1066e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:18.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 >> 172.21.15.78:0/3247117611 conn(0x7f122c0f0b70 msgr2=0x7f122c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:18.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 shutdown_connections 2024-01-21T08:09:18.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:18.768+0000 7f1230a4d700 1 -- 172.21.15.78:0/3247117611 wait complete. 2024-01-21T08:09:18.771 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:19.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:18.767473+0000 mon.smithi046 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.78:0/3247117611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:20.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:20.260 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:18.627760+0000 mgr.smithi046.lmxnwc (mgr.14164) 126 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3935302412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 msgr2=0x7fdf380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 --2- 172.21.15.78:0/3935302412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf380f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fdf2c009fb0 tx=0x7fdf2c009b30).stop 2024-01-21T08:09:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3935302412 shutdown_connections 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 --2- 172.21.15.78:0/3935302412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf380f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3935302412 >> 172.21.15.78:0/3935302412 conn(0x7fdf380f1c70 msgr2=0x7fdf380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3935302412 shutdown_connections 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3935302412 wait complete. 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 Processor -- start 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.432+0000 7fdf3c8d0700 1 -- start start 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf3c8d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf3c8d0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf38106c20 con 0x7fdf380f6510 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:36712/0 (socket says 172.21.15.78:36712) 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf3659c700 1 -- 172.21.15.78:0/3527226541 learned_addr learned my addr 172.21.15.78:0/3527226541 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf3659c700 1 -- 172.21.15.78:0/3527226541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c005040 con 0x7fdf380f6510 2024-01-21T08:09:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf3659c700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf38106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fdf2c013510 tx=0x7fdf2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.433+0000 7fdf26ffd700 1 -- 172.21.15.78:0/3527226541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2c009320 con 0x7fdf380f6510 2024-01-21T08:09:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.434+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf38106e20 con 0x7fdf380f6510 2024-01-21T08:09:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.434+0000 7fdf26ffd700 1 -- 172.21.15.78:0/3527226541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf2c013f00 con 0x7fdf380f6510 2024-01-21T08:09:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.434+0000 7fdf26ffd700 1 -- 172.21.15.78:0/3527226541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2c014d90 con 0x7fdf380f6510 2024-01-21T08:09:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.434+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf381072b0 con 0x7fdf380f6510 2024-01-21T08:09:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.434+0000 7fdf26ffd700 1 -- 172.21.15.78:0/3527226541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf2c0095f0 con 0x7fdf380f6510 2024-01-21T08:09:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.434+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38045ad0 con 0x7fdf380f6510 2024-01-21T08:09:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.434+0000 7fdf26ffd700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf2002ff60 0x7fdf20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.435+0000 7fdf26ffd700 1 -- 172.21.15.78:0/3527226541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf2c041010 con 0x7fdf380f6510 2024-01-21T08:09:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.435+0000 7fdf35d9b700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf2002ff60 0x7fdf20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.435+0000 7fdf35d9b700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf2002ff60 0x7fdf20032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fdf280096a0 tx=0x7fdf28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:21.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.440+0000 7fdf26ffd700 1 -- 172.21.15.78:0/3527226541 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf2c014420 con 0x7fdf380f6510 2024-01-21T08:09:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.648+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf380fbf60 con 0x7fdf380f6510 2024-01-21T08:09:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.649+0000 7fdf26ffd700 1 -- 172.21.15.78:0/3527226541 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf2c010230 con 0x7fdf380f6510 2024-01-21T08:09:21.651 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:21.651 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf2002ff60 msgr2=0x7fdf20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf2002ff60 0x7fdf20032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fdf280096a0 tx=0x7fdf28006b40).stop 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 msgr2=0x7fdf38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf38106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fdf2c013510 tx=0x7fdf2c00b040).stop 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 shutdown_connections 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdf2002ff60 0x7fdf20032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 --2- 172.21.15.78:0/3527226541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf380f6510 0x7fdf38106730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 >> 172.21.15.78:0/3527226541 conn(0x7fdf380f1c70 msgr2=0x7fdf380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 shutdown_connections 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:21.650+0000 7fdf3c8d0700 1 -- 172.21.15.78:0/3527226541 wait complete. 2024-01-21T08:09:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:20.742148+0000 mon.smithi046 (mon.0) 208 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:09:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:20.746006+0000 mon.smithi046 (mon.0) 209 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:09:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:20.628038+0000 mgr.smithi046.lmxnwc (mgr.14164) 127 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:21.649740+0000 mon.smithi046 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.78:0/3527226541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:23.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:23.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.207+0000 7f045d7af700 1 -- 172.21.15.78:0/998243184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 msgr2=0x7f04580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.207+0000 7f045d7af700 1 --2- 172.21.15.78:0/998243184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f04580f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f044c009fb0 tx=0x7f044c009b30).stop 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.207+0000 7f045d7af700 1 -- 172.21.15.78:0/998243184 shutdown_connections 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.207+0000 7f045d7af700 1 --2- 172.21.15.78:0/998243184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f04580f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.207+0000 7f045d7af700 1 -- 172.21.15.78:0/998243184 >> 172.21.15.78:0/998243184 conn(0x7f04580f0b70 msgr2=0x7f04580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.207+0000 7f045d7af700 1 -- 172.21.15.78:0/998243184 shutdown_connections 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.207+0000 7f045d7af700 1 -- 172.21.15.78:0/998243184 wait complete. 2024-01-21T08:09:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f045d7af700 1 Processor -- start 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f045d7af700 1 -- start start 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f045d7af700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f0458109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f045d7af700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458060220 con 0x7f04580f5410 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f0456ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f0458109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f0456ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f0458109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:39416/0 (socket says 172.21.15.78:39416) 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f0456ffd700 1 -- 172.21.15.78:0/2403598171 learned_addr learned my addr 172.21.15.78:0/2403598171 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.208+0000 7f0456ffd700 1 -- 172.21.15.78:0/2403598171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c005040 con 0x7f04580f5410 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.209+0000 7f0456ffd700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f0458109450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f044c014000 tx=0x7f044c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.209+0000 7f043f7fe700 1 -- 172.21.15.78:0/2403598171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f044c0095f0 con 0x7f04580f5410 2024-01-21T08:09:24.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.209+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0458060420 con 0x7f04580f5410 2024-01-21T08:09:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.209+0000 7f043f7fe700 1 -- 172.21.15.78:0/2403598171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f044c007e70 con 0x7f04580f5410 2024-01-21T08:09:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.209+0000 7f043f7fe700 1 -- 172.21.15.78:0/2403598171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f044c014ea0 con 0x7f04580f5410 2024-01-21T08:09:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.209+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0458109b00 con 0x7f04580f5410 2024-01-21T08:09:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.210+0000 7f043f7fe700 1 -- 172.21.15.78:0/2403598171 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f044c025070 con 0x7f04580f5410 2024-01-21T08:09:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.210+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0458059100 con 0x7f04580f5410 2024-01-21T08:09:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.210+0000 7f043f7fe700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f044002ff60 0x7f0440032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.210+0000 7f043f7fe700 1 -- 172.21.15.78:0/2403598171 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f044c040950 con 0x7f04580f5410 2024-01-21T08:09:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.210+0000 7f04567fc700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f044002ff60 0x7f0440032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.210+0000 7f04567fc700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f044002ff60 0x7f0440032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f04480096a0 tx=0x7f0448006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:24.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.214+0000 7f043f7fe700 1 -- 172.21.15.78:0/2403598171 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f044c026350 con 0x7f04580f5410 2024-01-21T08:09:24.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.423+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0458045ad0 con 0x7f04580f5410 2024-01-21T08:09:24.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.424+0000 7f043f7fe700 1 -- 172.21.15.78:0/2403598171 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f044c012330 con 0x7f04580f5410 2024-01-21T08:09:24.426 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:24.426 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f044002ff60 msgr2=0x7f0440032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f044002ff60 0x7f0440032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f04480096a0 tx=0x7f0448006b40).stop 2024-01-21T08:09:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 msgr2=0x7f0458109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f0458109450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f044c014000 tx=0x7f044c00b040).stop 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 shutdown_connections 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f044002ff60 0x7f0440032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 --2- 172.21.15.78:0/2403598171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04580f5410 0x7f0458109450 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 >> 172.21.15.78:0/2403598171 conn(0x7f04580f0b70 msgr2=0x7f04580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.425+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 shutdown_connections 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:24.426+0000 7f045d7af700 1 -- 172.21.15.78:0/2403598171 wait complete. 2024-01-21T08:09:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:22.628375+0000 mgr.smithi046.lmxnwc (mgr.14164) 128 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:24.424947+0000 mon.smithi046 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.78:0/2403598171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:25.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:25.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:26.583 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:24.628707+0000 mgr.smithi046.lmxnwc (mgr.14164) 129 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:26.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.972+0000 7f3ed027c700 1 -- 172.21.15.78:0/3706143011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 msgr2=0x7f3ec80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:26.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.972+0000 7f3ed027c700 1 --2- 172.21.15.78:0/3706143011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec80f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f3eb8009fb0 tx=0x7f3eb8009b30).stop 2024-01-21T08:09:26.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.972+0000 7f3ed027c700 1 -- 172.21.15.78:0/3706143011 shutdown_connections 2024-01-21T08:09:26.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.972+0000 7f3ed027c700 1 --2- 172.21.15.78:0/3706143011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec80f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:26.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.972+0000 7f3ed027c700 1 -- 172.21.15.78:0/3706143011 >> 172.21.15.78:0/3706143011 conn(0x7f3ec80f0b70 msgr2=0x7f3ec80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:26.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ed027c700 1 -- 172.21.15.78:0/3706143011 shutdown_connections 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ed027c700 1 -- 172.21.15.78:0/3706143011 wait complete. 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ed027c700 1 Processor -- start 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ed027c700 1 -- start start 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ed027c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec81093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ed027c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8060240 con 0x7f3ec80f5410 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ece018700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec81093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ece018700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec81093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:39434/0 (socket says 172.21.15.78:39434) 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.973+0000 7f3ece018700 1 -- 172.21.15.78:0/3994881612 learned_addr learned my addr 172.21.15.78:0/3994881612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.974+0000 7f3ece018700 1 -- 172.21.15.78:0/3994881612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb8005040 con 0x7f3ec80f5410 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.974+0000 7f3ece018700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec81093d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f3eb8014000 tx=0x7f3eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.974+0000 7f3ebeffd700 1 -- 172.21.15.78:0/3994881612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb80095f0 con 0x7f3ec80f5410 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.974+0000 7f3ebeffd700 1 -- 172.21.15.78:0/3994881612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3eb8007e70 con 0x7f3ec80f5410 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.974+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec8060440 con 0x7f3ec80f5410 2024-01-21T08:09:26.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.974+0000 7f3ebeffd700 1 -- 172.21.15.78:0/3994881612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb8014ea0 con 0x7f3ec80f5410 2024-01-21T08:09:26.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.974+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec8109a80 con 0x7f3ec80f5410 2024-01-21T08:09:26.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.975+0000 7f3ebeffd700 1 -- 172.21.15.78:0/3994881612 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3eb8025070 con 0x7f3ec80f5410 2024-01-21T08:09:26.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.975+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8045ad0 con 0x7f3ec80f5410 2024-01-21T08:09:26.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.975+0000 7f3ebeffd700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3eb402ff60 0x7f3eb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:26.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.975+0000 7f3ebeffd700 1 -- 172.21.15.78:0/3994881612 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3eb8040a30 con 0x7f3ec80f5410 2024-01-21T08:09:26.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.975+0000 7f3ecd817700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3eb402ff60 0x7f3eb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:26.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.975+0000 7f3ecd817700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3eb402ff60 0x7f3eb4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3ec40096a0 tx=0x7f3ec4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:26.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:26.980+0000 7f3ebeffd700 1 -- 172.21.15.78:0/3994881612 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3eb800f4e0 con 0x7f3ec80f5410 2024-01-21T08:09:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.189+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ec810a2a0 con 0x7f3ec80f5410 2024-01-21T08:09:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.190+0000 7f3ebeffd700 1 -- 172.21.15.78:0/3994881612 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3eb800f300 con 0x7f3ec80f5410 2024-01-21T08:09:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:27.192 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3eb402ff60 msgr2=0x7f3eb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3eb402ff60 0x7f3eb4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3ec40096a0 tx=0x7f3ec4006b40).stop 2024-01-21T08:09:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 msgr2=0x7f3ec81093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec81093d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f3eb8014000 tx=0x7f3eb800b040).stop 2024-01-21T08:09:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 shutdown_connections 2024-01-21T08:09:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3eb402ff60 0x7f3eb4032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 --2- 172.21.15.78:0/3994881612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec80f5410 0x7f3ec81093d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 >> 172.21.15.78:0/3994881612 conn(0x7f3ec80f0b70 msgr2=0x7f3ec80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 shutdown_connections 2024-01-21T08:09:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:27.191+0000 7f3ed027c700 1 -- 172.21.15.78:0/3994881612 wait complete. 2024-01-21T08:09:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:27.190825+0000 mon.smithi046 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.78:0/3994881612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:28.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:28.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:26.628978+0000 mgr.smithi046.lmxnwc (mgr.14164) 130 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:29.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/3658772916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 msgr2=0x7f9e080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 --2- 172.21.15.78:0/3658772916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e080f66c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9e04009fb0 tx=0x7f9e04009b30).stop 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/3658772916 shutdown_connections 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 --2- 172.21.15.78:0/3658772916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e080f66c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/3658772916 >> 172.21.15.78:0/3658772916 conn(0x7f9e080f1aa0 msgr2=0x7f9e080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/3658772916 shutdown_connections 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/3658772916 wait complete. 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 Processor -- start 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.816+0000 7f9e0ed9a700 1 -- start start 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0ed9a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e081064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0ed9a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e081069a0 con 0x7f9e080f62a0 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0dd98700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e081064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0dd98700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e081064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:39448/0 (socket says 172.21.15.78:39448) 2024-01-21T08:09:29.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0dd98700 1 -- 172.21.15.78:0/1500149211 learned_addr learned my addr 172.21.15.78:0/1500149211 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0dd98700 1 -- 172.21.15.78:0/1500149211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04005040 con 0x7f9e080f62a0 2024-01-21T08:09:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0dd98700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e081064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f9e04013510 tx=0x7f9e0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9dfe7fc700 1 -- 172.21.15.78:0/1500149211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e04009320 con 0x7f9e080f62a0 2024-01-21T08:09:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9dfe7fc700 1 -- 172.21.15.78:0/1500149211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e04013f00 con 0x7f9e080f62a0 2024-01-21T08:09:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08106ba0 con 0x7f9e080f62a0 2024-01-21T08:09:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9dfe7fc700 1 -- 172.21.15.78:0/1500149211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e04014dc0 con 0x7f9e080f62a0 2024-01-21T08:09:29.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.817+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e08107030 con 0x7f9e080f62a0 2024-01-21T08:09:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.818+0000 7f9dfe7fc700 1 -- 172.21.15.78:0/1500149211 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e0401d3d0 con 0x7f9e080f62a0 2024-01-21T08:09:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.818+0000 7f9dfe7fc700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9df402ff60 0x7f9df4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.818+0000 7f9dfe7fc700 1 -- 172.21.15.78:0/1500149211 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e04041080 con 0x7f9e080f62a0 2024-01-21T08:09:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.818+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e08045ad0 con 0x7f9e080f62a0 2024-01-21T08:09:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.819+0000 7f9e0d597700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9df402ff60 0x7f9df4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:29.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.819+0000 7f9e0d597700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9df402ff60 0x7f9df4032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9df80096a0 tx=0x7f9df8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:29.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:29.823+0000 7f9dfe7fc700 1 -- 172.21.15.78:0/1500149211 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e04012920 con 0x7f9e080f62a0 2024-01-21T08:09:30.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.028+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e08107d50 con 0x7f9e080f62a0 2024-01-21T08:09:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.028+0000 7f9dfe7fc700 1 -- 172.21.15.78:0/1500149211 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e0402b020 con 0x7f9e080f62a0 2024-01-21T08:09:30.030 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:30.030 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9df402ff60 msgr2=0x7f9df4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9df402ff60 0x7f9df4032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9df80096a0 tx=0x7f9df8006b40).stop 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 msgr2=0x7f9e081064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e081064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f9e04013510 tx=0x7f9e0400b040).stop 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 shutdown_connections 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9df402ff60 0x7f9df4032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 --2- 172.21.15.78:0/1500149211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080f62a0 0x7f9e081064b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 >> 172.21.15.78:0/1500149211 conn(0x7f9e080f1aa0 msgr2=0x7f9e080f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 shutdown_connections 2024-01-21T08:09:30.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:30.030+0000 7f9e0ed9a700 1 -- 172.21.15.78:0/1500149211 wait complete. 2024-01-21T08:09:30.033 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:28.629323+0000 mgr.smithi046.lmxnwc (mgr.14164) 131 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:30.029430+0000 mon.smithi046 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.78:0/1500149211' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:31.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:31.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:32.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 -- 172.21.15.78:0/2505800927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 msgr2=0x7fdcd0069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 --2- 172.21.15.78:0/2505800927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0069e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fdcc4009fb0 tx=0x7fdcc4009b30).stop 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 -- 172.21.15.78:0/2505800927 shutdown_connections 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 --2- 172.21.15.78:0/2505800927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0069e60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 -- 172.21.15.78:0/2505800927 >> 172.21.15.78:0/2505800927 conn(0x7fdcd00f1aa0 msgr2=0x7fdcd00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 -- 172.21.15.78:0/2505800927 shutdown_connections 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 -- 172.21.15.78:0/2505800927 wait complete. 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.689+0000 7fdcd51a2700 1 Processor -- start 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdcd51a2700 1 -- start start 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdcd51a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0108730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdcd51a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd0108c20 con 0x7fdcd006b8c0 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdccffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0108730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:32.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdccffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0108730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35568/0 (socket says 172.21.15.78:35568) 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdccffff700 1 -- 172.21.15.78:0/839954198 learned_addr learned my addr 172.21.15.78:0/839954198 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdccffff700 1 -- 172.21.15.78:0/839954198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc4005040 con 0x7fdcd006b8c0 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.690+0000 7fdccffff700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0108730 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdcc4013510 tx=0x7fdcc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.691+0000 7fdcccff9700 1 -- 172.21.15.78:0/839954198 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc4009320 con 0x7fdcd006b8c0 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.691+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd0108e20 con 0x7fdcd006b8c0 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.691+0000 7fdcccff9700 1 -- 172.21.15.78:0/839954198 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcc4013f00 con 0x7fdcd006b8c0 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.691+0000 7fdcccff9700 1 -- 172.21.15.78:0/839954198 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc4014d90 con 0x7fdcd006b8c0 2024-01-21T08:09:32.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.691+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd01092b0 con 0x7fdcd006b8c0 2024-01-21T08:09:32.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.691+0000 7fdcccff9700 1 -- 172.21.15.78:0/839954198 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcc40263a0 con 0x7fdcd006b8c0 2024-01-21T08:09:32.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.692+0000 7fdcccff9700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdcb802ff60 0x7fdcb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:32.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.692+0000 7fdcccff9700 1 -- 172.21.15.78:0/839954198 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcc40410d0 con 0x7fdcd006b8c0 2024-01-21T08:09:32.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.692+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd0045ad0 con 0x7fdcd006b8c0 2024-01-21T08:09:32.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.692+0000 7fdccf7fe700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdcb802ff60 0x7fdcb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:32.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.692+0000 7fdccf7fe700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdcb802ff60 0x7fdcb8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdcc00096a0 tx=0x7fdcc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:32.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.697+0000 7fdcccff9700 1 -- 172.21.15.78:0/839954198 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcc4012920 con 0x7fdcd006b8c0 2024-01-21T08:09:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:30.629725+0000 mgr.smithi046.lmxnwc (mgr.14164) 132 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:32.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.898+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcd0109f30 con 0x7fdcd006b8c0 2024-01-21T08:09:32.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.899+0000 7fdcccff9700 1 -- 172.21.15.78:0/839954198 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcc4012330 con 0x7fdcd006b8c0 2024-01-21T08:09:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:32.901 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:32.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdcb802ff60 msgr2=0x7fdcb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdcb802ff60 0x7fdcb8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdcc00096a0 tx=0x7fdcc0006b40).stop 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 msgr2=0x7fdcd0108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0108730 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdcc4013510 tx=0x7fdcc400b040).stop 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 shutdown_connections 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdcb802ff60 0x7fdcb8032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 --2- 172.21.15.78:0/839954198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd006b8c0 0x7fdcd0108730 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 >> 172.21.15.78:0/839954198 conn(0x7fdcd00f1aa0 msgr2=0x7fdcd00f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 shutdown_connections 2024-01-21T08:09:32.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:32.901+0000 7fdcd51a2700 1 -- 172.21.15.78:0/839954198 wait complete. 2024-01-21T08:09:32.904 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:33.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:32.900004+0000 mon.smithi046 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.78:0/839954198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:34.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:34.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:34.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:32.630002+0000 mgr.smithi046.lmxnwc (mgr.14164) 133 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:35.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca27fc700 1 -- 172.21.15.78:0/668679845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc980138a0 con 0x7fdca40f47c0 2024-01-21T08:09:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca9ef8700 1 -- 172.21.15.78:0/668679845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 msgr2=0x7fdca40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca9ef8700 1 --2- 172.21.15.78:0/668679845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca40f4be0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fdc98009fb0 tx=0x7fdc98009b30).stop 2024-01-21T08:09:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca9ef8700 1 -- 172.21.15.78:0/668679845 shutdown_connections 2024-01-21T08:09:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca9ef8700 1 --2- 172.21.15.78:0/668679845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca40f4be0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca9ef8700 1 -- 172.21.15.78:0/668679845 >> 172.21.15.78:0/668679845 conn(0x7fdca40f02e0 msgr2=0x7fdca40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca9ef8700 1 -- 172.21.15.78:0/668679845 shutdown_connections 2024-01-21T08:09:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.433+0000 7fdca9ef8700 1 -- 172.21.15.78:0/668679845 wait complete. 2024-01-21T08:09:35.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.434+0000 7fdca9ef8700 1 Processor -- start 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.434+0000 7fdca9ef8700 1 -- start start 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.435+0000 7fdca9ef8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca4108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.435+0000 7fdca9ef8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4108df0 con 0x7fdca40f47c0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.435+0000 7fdca37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca4108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.435+0000 7fdca37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca4108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35588/0 (socket says 172.21.15.78:35588) 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.435+0000 7fdca37fe700 1 -- 172.21.15.78:0/3086605360 learned_addr learned my addr 172.21.15.78:0/3086605360 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.435+0000 7fdca37fe700 1 -- 172.21.15.78:0/3086605360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc98005040 con 0x7fdca40f47c0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.435+0000 7fdca37fe700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca4108900 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdc98013510 tx=0x7fdc9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.436+0000 7fdc8bfff700 1 -- 172.21.15.78:0/3086605360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc98009320 con 0x7fdca40f47c0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.436+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca4108ff0 con 0x7fdca40f47c0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.436+0000 7fdc8bfff700 1 -- 172.21.15.78:0/3086605360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc98013f00 con 0x7fdca40f47c0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.436+0000 7fdc8bfff700 1 -- 172.21.15.78:0/3086605360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc98014d90 con 0x7fdca40f47c0 2024-01-21T08:09:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.436+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4109480 con 0x7fdca40f47c0 2024-01-21T08:09:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.437+0000 7fdc8bfff700 1 -- 172.21.15.78:0/3086605360 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc98014420 con 0x7fdca40f47c0 2024-01-21T08:09:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.437+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca4045ad0 con 0x7fdca40f47c0 2024-01-21T08:09:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.437+0000 7fdc8bfff700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdc8c02ff60 0x7fdc8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.437+0000 7fdc8bfff700 1 -- 172.21.15.78:0/3086605360 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc98041000 con 0x7fdca40f47c0 2024-01-21T08:09:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.437+0000 7fdca2ffd700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdc8c02ff60 0x7fdc8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.437+0000 7fdca2ffd700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdc8c02ff60 0x7fdc8c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fdc940096a0 tx=0x7fdc94006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.442+0000 7fdc8bfff700 1 -- 172.21.15.78:0/3086605360 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc980095f0 con 0x7fdca40f47c0 2024-01-21T08:09:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.649+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdca40f6b20 con 0x7fdca40f47c0 2024-01-21T08:09:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.651+0000 7fdc8bfff700 1 -- 172.21.15.78:0/3086605360 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc9801db70 con 0x7fdca40f47c0 2024-01-21T08:09:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:35.653 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:35.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.651+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdc8c02ff60 msgr2=0x7fdc8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:35.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.651+0000 7fdca9ef8700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdc8c02ff60 0x7fdc8c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fdc940096a0 tx=0x7fdc94006b40).stop 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 msgr2=0x7fdca4108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca4108900 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdc98013510 tx=0x7fdc9800b040).stop 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 shutdown_connections 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fdc8c02ff60 0x7fdc8c032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 --2- 172.21.15.78:0/3086605360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca40f47c0 0x7fdca4108900 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 >> 172.21.15.78:0/3086605360 conn(0x7fdca40f02e0 msgr2=0x7fdca40f10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 shutdown_connections 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:35.652+0000 7fdca9ef8700 1 -- 172.21.15.78:0/3086605360 wait complete. 2024-01-21T08:09:35.655 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:36.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:34.630253+0000 mgr.smithi046.lmxnwc (mgr.14164) 134 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:36.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:35.651952+0000 mon.smithi046 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.78:0/3086605360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:37.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:37.047 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:38.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.215+0000 7f4a8b747700 1 -- 172.21.15.78:0/1656913264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 msgr2=0x7f4a840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.215+0000 7f4a8b747700 1 --2- 172.21.15.78:0/1656913264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a840f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f4a74009fb0 tx=0x7f4a74009b30).stop 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.215+0000 7f4a8b747700 1 -- 172.21.15.78:0/1656913264 shutdown_connections 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.215+0000 7f4a8b747700 1 --2- 172.21.15.78:0/1656913264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a840f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.215+0000 7f4a8b747700 1 -- 172.21.15.78:0/1656913264 >> 172.21.15.78:0/1656913264 conn(0x7f4a840f0b70 msgr2=0x7f4a840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.215+0000 7f4a8b747700 1 -- 172.21.15.78:0/1656913264 shutdown_connections 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.215+0000 7f4a8b747700 1 -- 172.21.15.78:0/1656913264 wait complete. 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a8b747700 1 Processor -- start 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a8b747700 1 -- start start 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a8b747700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a84109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a8b747700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84060220 con 0x7f4a840f5410 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a894e3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a84109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a894e3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a84109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35610/0 (socket says 172.21.15.78:35610) 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a894e3700 1 -- 172.21.15.78:0/772802745 learned_addr learned my addr 172.21.15.78:0/772802745 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a894e3700 1 -- 172.21.15.78:0/772802745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74005040 con 0x7f4a840f5410 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.216+0000 7f4a894e3700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a84109420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f4a74014000 tx=0x7f4a7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.217+0000 7f4a79ffb700 1 -- 172.21.15.78:0/772802745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a740095f0 con 0x7f4a840f5410 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.217+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a84060420 con 0x7f4a840f5410 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.217+0000 7f4a79ffb700 1 -- 172.21.15.78:0/772802745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a74007e70 con 0x7f4a840f5410 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.217+0000 7f4a79ffb700 1 -- 172.21.15.78:0/772802745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a74014ea0 con 0x7f4a840f5410 2024-01-21T08:09:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.217+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a84109ad0 con 0x7f4a840f5410 2024-01-21T08:09:38.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.218+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a84059100 con 0x7f4a840f5410 2024-01-21T08:09:38.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.218+0000 7f4a79ffb700 1 -- 172.21.15.78:0/772802745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a74025070 con 0x7f4a840f5410 2024-01-21T08:09:38.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.218+0000 7f4a79ffb700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:38.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.218+0000 7f4a79ffb700 1 -- 172.21.15.78:0/772802745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a74040980 con 0x7f4a840f5410 2024-01-21T08:09:38.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.218+0000 7f4a88ce2700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:38.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.219+0000 7f4a88ce2700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4a7002ff60 0x7f4a70032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f4a800096a0 tx=0x7f4a80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.223+0000 7f4a79ffb700 1 -- 172.21.15.78:0/772802745 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a740079c0 con 0x7f4a840f5410 2024-01-21T08:09:38.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.430+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a84045ad0 con 0x7f4a840f5410 2024-01-21T08:09:38.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.431+0000 7f4a79ffb700 1 -- 172.21.15.78:0/772802745 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a7400f300 con 0x7f4a840f5410 2024-01-21T08:09:38.434 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:38.434 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:38.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4a7002ff60 msgr2=0x7f4a70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4a7002ff60 0x7f4a70032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f4a800096a0 tx=0x7f4a80006b40).stop 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 msgr2=0x7f4a84109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a84109420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f4a74014000 tx=0x7f4a7400b040).stop 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 shutdown_connections 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 --2- 172.21.15.78:0/772802745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a840f5410 0x7f4a84109420 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 >> 172.21.15.78:0/772802745 conn(0x7f4a840f0b70 msgr2=0x7f4a840f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 shutdown_connections 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:38.432+0000 7f4a8b747700 1 -- 172.21.15.78:0/772802745 wait complete. 2024-01-21T08:09:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:36.630600+0000 mgr.smithi046.lmxnwc (mgr.14164) 135 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:39.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:38.432828+0000 mon.smithi046 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.78:0/772802745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:39.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:39.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:38.630875+0000 mgr.smithi046.lmxnwc (mgr.14164) 136 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:40.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.983+0000 7f368dd8a700 1 -- 172.21.15.78:0/738814353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 msgr2=0x7f36880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.983+0000 7f368dd8a700 1 --2- 172.21.15.78:0/738814353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f36880f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f367c009fb0 tx=0x7f367c009b30).stop 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.983+0000 7f368dd8a700 1 -- 172.21.15.78:0/738814353 shutdown_connections 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.983+0000 7f368dd8a700 1 --2- 172.21.15.78:0/738814353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f36880f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.983+0000 7f368dd8a700 1 -- 172.21.15.78:0/738814353 >> 172.21.15.78:0/738814353 conn(0x7f36880f0b70 msgr2=0x7f36880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.983+0000 7f368dd8a700 1 -- 172.21.15.78:0/738814353 shutdown_connections 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.983+0000 7f368dd8a700 1 -- 172.21.15.78:0/738814353 wait complete. 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f368dd8a700 1 Processor -- start 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f368dd8a700 1 -- start start 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f368dd8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f3688109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:40.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f368dd8a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3688060220 con 0x7f36880f5410 2024-01-21T08:09:40.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f36877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f3688109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:40.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f36877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f3688109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:35640/0 (socket says 172.21.15.78:35640) 2024-01-21T08:09:40.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f36877fe700 1 -- 172.21.15.78:0/1566953052 learned_addr learned my addr 172.21.15.78:0/1566953052 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:40.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.984+0000 7f36877fe700 1 -- 172.21.15.78:0/1566953052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c005040 con 0x7f36880f5410 2024-01-21T08:09:40.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.985+0000 7f36877fe700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f3688109450 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f367c013510 tx=0x7f367c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:40.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.985+0000 7f366ffff700 1 -- 172.21.15.78:0/1566953052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f367c009320 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.985+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3688060420 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.985+0000 7f366ffff700 1 -- 172.21.15.78:0/1566953052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f367c013f00 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.985+0000 7f366ffff700 1 -- 172.21.15.78:0/1566953052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f367c014d90 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.985+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3688109b00 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.986+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3688059100 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.986+0000 7f366ffff700 1 -- 172.21.15.78:0/1566953052 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f367c0095f0 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.986+0000 7f366ffff700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f367002ff60 0x7f3670032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.986+0000 7f366ffff700 1 -- 172.21.15.78:0/1566953052 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f367c040fe0 con 0x7f36880f5410 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.986+0000 7f3686ffd700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f367002ff60 0x7f3670032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:40.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.986+0000 7f3686ffd700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f367002ff60 0x7f3670032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f36780096a0 tx=0x7f3678006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:40.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:40.991+0000 7f366ffff700 1 -- 172.21.15.78:0/1566953052 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f367c01d7f0 con 0x7f36880f5410 2024-01-21T08:09:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.200+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3688045ad0 con 0x7f36880f5410 2024-01-21T08:09:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.201+0000 7f366ffff700 1 -- 172.21.15.78:0/1566953052 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f367c010230 con 0x7f36880f5410 2024-01-21T08:09:41.203 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:41.203 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f367002ff60 msgr2=0x7f3670032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f367002ff60 0x7f3670032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f36780096a0 tx=0x7f3678006b40).stop 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 msgr2=0x7f3688109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f3688109450 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f367c013510 tx=0x7f367c00b040).stop 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 shutdown_connections 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f367002ff60 0x7f3670032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 --2- 172.21.15.78:0/1566953052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36880f5410 0x7f3688109450 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 >> 172.21.15.78:0/1566953052 conn(0x7f36880f0b70 msgr2=0x7f36880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 shutdown_connections 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:41.203+0000 7f368dd8a700 1 -- 172.21.15.78:0/1566953052 wait complete. 2024-01-21T08:09:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:41.201981+0000 mon.smithi046 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.78:0/1566953052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:42.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:42.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:40.631107+0000 mgr.smithi046.lmxnwc (mgr.14164) 137 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:43.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 -- 172.21.15.78:0/1153086842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 msgr2=0x7f630c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:43.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 --2- 172.21.15.78:0/1153086842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c0f66c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f62fc009fb0 tx=0x7f62fc009b30).stop 2024-01-21T08:09:43.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 -- 172.21.15.78:0/1153086842 shutdown_connections 2024-01-21T08:09:43.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 --2- 172.21.15.78:0/1153086842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c0f66c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:43.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 -- 172.21.15.78:0/1153086842 >> 172.21.15.78:0/1153086842 conn(0x7f630c0f1aa0 msgr2=0x7f630c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:43.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 -- 172.21.15.78:0/1153086842 shutdown_connections 2024-01-21T08:09:43.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 -- 172.21.15.78:0/1153086842 wait complete. 2024-01-21T08:09:43.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 Processor -- start 2024-01-21T08:09:43.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.816+0000 7f6312aa7700 1 -- start start 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6312aa7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6312aa7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c1069a0 con 0x7f630c0f62a0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6311aa5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6311aa5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52676/0 (socket says 172.21.15.78:52676) 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6311aa5700 1 -- 172.21.15.78:0/3176184406 learned_addr learned my addr 172.21.15.78:0/3176184406 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6311aa5700 1 -- 172.21.15.78:0/3176184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc005040 con 0x7f630c0f62a0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6311aa5700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c1064b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f62fc013510 tx=0x7f62fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f63027fc700 1 -- 172.21.15.78:0/3176184406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc009320 con 0x7f630c0f62a0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f63027fc700 1 -- 172.21.15.78:0/3176184406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62fc013f00 con 0x7f630c0f62a0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c106ba0 con 0x7f630c0f62a0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f63027fc700 1 -- 172.21.15.78:0/3176184406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc014dc0 con 0x7f630c0f62a0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.817+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c107030 con 0x7f630c0f62a0 2024-01-21T08:09:43.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.818+0000 7f63027fc700 1 -- 172.21.15.78:0/3176184406 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62fc0095f0 con 0x7f630c0f62a0 2024-01-21T08:09:43.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.818+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c045ad0 con 0x7f630c0f62a0 2024-01-21T08:09:43.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.818+0000 7f63027fc700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f62f802ff60 0x7f62f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:43.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.818+0000 7f63027fc700 1 -- 172.21.15.78:0/3176184406 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62fc041000 con 0x7f630c0f62a0 2024-01-21T08:09:43.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.819+0000 7f63112a4700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f62f802ff60 0x7f62f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:43.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.819+0000 7f63112a4700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f62f802ff60 0x7f62f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f63080096a0 tx=0x7f6308006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:43.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:43.823+0000 7f63027fc700 1 -- 172.21.15.78:0/3176184406 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62fc014420 con 0x7f630c0f62a0 2024-01-21T08:09:44.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.026+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f630c107ee0 con 0x7f630c0f62a0 2024-01-21T08:09:44.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.027+0000 7f63027fc700 1 -- 172.21.15.78:0/3176184406 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62fc010230 con 0x7f630c0f62a0 2024-01-21T08:09:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:44.029 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f62f802ff60 msgr2=0x7f62f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f62f802ff60 0x7f62f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f63080096a0 tx=0x7f6308006b40).stop 2024-01-21T08:09:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 msgr2=0x7f630c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c1064b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f62fc013510 tx=0x7f62fc00b040).stop 2024-01-21T08:09:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 shutdown_connections 2024-01-21T08:09:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f62f802ff60 0x7f62f8032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 --2- 172.21.15.78:0/3176184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f630c0f62a0 0x7f630c1064b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 >> 172.21.15.78:0/3176184406 conn(0x7f630c0f1aa0 msgr2=0x7f630c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 shutdown_connections 2024-01-21T08:09:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:44.029+0000 7f6312aa7700 1 -- 172.21.15.78:0/3176184406 wait complete. 2024-01-21T08:09:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:44.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:42.631442+0000 mgr.smithi046.lmxnwc (mgr.14164) 138 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:44.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:44.028275+0000 mon.smithi046 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.78:0/3176184406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:45.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:45.422 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:46.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 -- 172.21.15.78:0/2447031575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 msgr2=0x7f12b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 --2- 172.21.15.78:0/2447031575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b40f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f12a8009fb0 tx=0x7f12a8009b30).stop 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 -- 172.21.15.78:0/2447031575 shutdown_connections 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 --2- 172.21.15.78:0/2447031575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b40f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 -- 172.21.15.78:0/2447031575 >> 172.21.15.78:0/2447031575 conn(0x7f12b40f0b70 msgr2=0x7f12b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 -- 172.21.15.78:0/2447031575 shutdown_connections 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 -- 172.21.15.78:0/2447031575 wait complete. 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 Processor -- start 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.554+0000 7f12bac22700 1 -- start start 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.555+0000 7f12bac22700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.555+0000 7f12bac22700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b4105b20 con 0x7f12b40f5410 2024-01-21T08:09:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.555+0000 7f12b89be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.555+0000 7f12b89be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52698/0 (socket says 172.21.15.78:52698) 2024-01-21T08:09:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.555+0000 7f12b89be700 1 -- 172.21.15.78:0/821366645 learned_addr learned my addr 172.21.15.78:0/821366645 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.556+0000 7f12b89be700 1 -- 172.21.15.78:0/821366645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8005040 con 0x7f12b40f5410 2024-01-21T08:09:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.556+0000 7f12b89be700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b4105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f12a8013510 tx=0x7f12a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.556+0000 7f12b17fa700 1 -- 172.21.15.78:0/821366645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a8009320 con 0x7f12b40f5410 2024-01-21T08:09:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.556+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b4105d20 con 0x7f12b40f5410 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.556+0000 7f12b17fa700 1 -- 172.21.15.78:0/821366645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12a8013f00 con 0x7f12b40f5410 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.556+0000 7f12b17fa700 1 -- 172.21.15.78:0/821366645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a8014d90 con 0x7f12b40f5410 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.556+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b41061b0 con 0x7f12b40f5410 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.557+0000 7f12b17fa700 1 -- 172.21.15.78:0/821366645 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12a80095f0 con 0x7f12b40f5410 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.557+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4045ad0 con 0x7f12b40f5410 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.557+0000 7f12b17fa700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12a402ff60 0x7f12a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.557+0000 7f12b17fa700 1 -- 172.21.15.78:0/821366645 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12a8040fe0 con 0x7f12b40f5410 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.557+0000 7f12b3fff700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12a402ff60 0x7f12a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.557+0000 7f12b3fff700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12a402ff60 0x7f12a4032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f12a00096a0 tx=0x7f12a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:46.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.561+0000 7f12b17fa700 1 -- 172.21.15.78:0/821366645 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12a801d7f0 con 0x7f12b40f5410 2024-01-21T08:09:46.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:44.631838+0000 mgr.smithi046.lmxnwc (mgr.14164) 139 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:46.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.770+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12b40fae60 con 0x7f12b40f5410 2024-01-21T08:09:46.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.771+0000 7f12b17fa700 1 -- 172.21.15.78:0/821366645 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12a8010230 con 0x7f12b40f5410 2024-01-21T08:09:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:46.777 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12a402ff60 msgr2=0x7f12a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12a402ff60 0x7f12a4032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f12a00096a0 tx=0x7f12a0006b40).stop 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 msgr2=0x7f12b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b4105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f12a8013510 tx=0x7f12a800b040).stop 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 shutdown_connections 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f12a402ff60 0x7f12a4032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 --2- 172.21.15.78:0/821366645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b40f5410 0x7f12b4105630 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 >> 172.21.15.78:0/821366645 conn(0x7f12b40f0b70 msgr2=0x7f12b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 shutdown_connections 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:46.773+0000 7f12bac22700 1 -- 172.21.15.78:0/821366645 wait complete. 2024-01-21T08:09:46.778 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:46.772180+0000 mon.smithi046 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.78:0/821366645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:48.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:48.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:46.632135+0000 mgr.smithi046.lmxnwc (mgr.14164) 140 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.390+0000 7f34c2aec700 1 -- 172.21.15.78:0/1477313884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 msgr2=0x7f34bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.390+0000 7f34c2aec700 1 --2- 172.21.15.78:0/1477313884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc0f66c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f34ac009fb0 tx=0x7f34ac009b30).stop 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.390+0000 7f34c2aec700 1 -- 172.21.15.78:0/1477313884 shutdown_connections 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.390+0000 7f34c2aec700 1 --2- 172.21.15.78:0/1477313884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc0f66c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.390+0000 7f34c2aec700 1 -- 172.21.15.78:0/1477313884 >> 172.21.15.78:0/1477313884 conn(0x7f34bc0f1aa0 msgr2=0x7f34bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.390+0000 7f34c2aec700 1 -- 172.21.15.78:0/1477313884 shutdown_connections 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.390+0000 7f34c2aec700 1 -- 172.21.15.78:0/1477313884 wait complete. 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c2aec700 1 Processor -- start 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c2aec700 1 -- start start 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c2aec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c2aec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc1069a0 con 0x7f34bc0f62a0 2024-01-21T08:09:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c1aea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c1aea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52718/0 (socket says 172.21.15.78:52718) 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c1aea700 1 -- 172.21.15.78:0/3799723233 learned_addr learned my addr 172.21.15.78:0/3799723233 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c1aea700 1 -- 172.21.15.78:0/3799723233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac005040 con 0x7f34bc0f62a0 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.391+0000 7f34c1aea700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc1064b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f34ac013510 tx=0x7f34ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.392+0000 7f34b27fc700 1 -- 172.21.15.78:0/3799723233 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ac009320 con 0x7f34bc0f62a0 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.392+0000 7f34b27fc700 1 -- 172.21.15.78:0/3799723233 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34ac013f00 con 0x7f34bc0f62a0 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.392+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc106ba0 con 0x7f34bc0f62a0 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.392+0000 7f34b27fc700 1 -- 172.21.15.78:0/3799723233 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ac014dc0 con 0x7f34bc0f62a0 2024-01-21T08:09:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.392+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc107030 con 0x7f34bc0f62a0 2024-01-21T08:09:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.392+0000 7f34b27fc700 1 -- 172.21.15.78:0/3799723233 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34ac0095f0 con 0x7f34bc0f62a0 2024-01-21T08:09:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.392+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34bc045ad0 con 0x7f34bc0f62a0 2024-01-21T08:09:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.393+0000 7f34b27fc700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f34a802ff60 0x7f34a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.393+0000 7f34b27fc700 1 -- 172.21.15.78:0/3799723233 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34ac041000 con 0x7f34bc0f62a0 2024-01-21T08:09:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.393+0000 7f34c12e9700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f34a802ff60 0x7f34a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.393+0000 7f34c12e9700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f34a802ff60 0x7f34a8032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f34b80096a0 tx=0x7f34b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.397+0000 7f34b27fc700 1 -- 172.21.15.78:0/3799723233 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34ac014420 con 0x7f34bc0f62a0 2024-01-21T08:09:49.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.601+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34bc107ee0 con 0x7f34bc0f62a0 2024-01-21T08:09:49.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.601+0000 7f34b27fc700 1 -- 172.21.15.78:0/3799723233 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34ac010230 con 0x7f34bc0f62a0 2024-01-21T08:09:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:49.603 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f34a802ff60 msgr2=0x7f34a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f34a802ff60 0x7f34a8032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f34b80096a0 tx=0x7f34b8006b40).stop 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 msgr2=0x7f34bc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc1064b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f34ac013510 tx=0x7f34ac00b040).stop 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 shutdown_connections 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f34a802ff60 0x7f34a8032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 --2- 172.21.15.78:0/3799723233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34bc0f62a0 0x7f34bc1064b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 >> 172.21.15.78:0/3799723233 conn(0x7f34bc0f1aa0 msgr2=0x7f34bc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:49.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 shutdown_connections 2024-01-21T08:09:49.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:49.603+0000 7f34c2aec700 1 -- 172.21.15.78:0/3799723233 wait complete. 2024-01-21T08:09:49.606 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:48.632471+0000 mgr.smithi046.lmxnwc (mgr.14164) 141 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:49.602336+0000 mon.smithi046 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.78:0/3799723233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:51.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:51.015 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:52.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 -- 172.21.15.78:0/827683557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 msgr2=0x7f02b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 --2- 172.21.15.78:0/827683557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40f55c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f02a4009fb0 tx=0x7f02a4009b30).stop 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 -- 172.21.15.78:0/827683557 shutdown_connections 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 --2- 172.21.15.78:0/827683557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40f55c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 -- 172.21.15.78:0/827683557 >> 172.21.15.78:0/827683557 conn(0x7f02b40f09a0 msgr2=0x7f02b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 -- 172.21.15.78:0/827683557 shutdown_connections 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 -- 172.21.15.78:0/827683557 wait complete. 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 Processor -- start 2024-01-21T08:09:52.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.223+0000 7f02bb080700 1 -- start start 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02bb080700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02bb080700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40fb5a0 con 0x7f02b40f51a0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02ba07e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02ba07e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52736/0 (socket says 172.21.15.78:52736) 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02ba07e700 1 -- 172.21.15.78:0/3163842338 learned_addr learned my addr 172.21.15.78:0/3163842338 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02ba07e700 1 -- 172.21.15.78:0/3163842338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a4005040 con 0x7f02b40f51a0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02ba07e700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40fd120 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f02a4013510 tx=0x7f02a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02aaffd700 1 -- 172.21.15.78:0/3163842338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a4009320 con 0x7f02b40f51a0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b40fb740 con 0x7f02b40f51a0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02aaffd700 1 -- 172.21.15.78:0/3163842338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02a4013f00 con 0x7f02b40f51a0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02aaffd700 1 -- 172.21.15.78:0/3163842338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a4014dc0 con 0x7f02b40f51a0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.224+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b40fbbd0 con 0x7f02b40f51a0 2024-01-21T08:09:52.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.225+0000 7f02aaffd700 1 -- 172.21.15.78:0/3163842338 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02a40095f0 con 0x7f02b40f51a0 2024-01-21T08:09:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.225+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b4059100 con 0x7f02b40f51a0 2024-01-21T08:09:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.225+0000 7f02aaffd700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f02a002ff60 0x7f02a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.225+0000 7f02aaffd700 1 -- 172.21.15.78:0/3163842338 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02a4041030 con 0x7f02b40f51a0 2024-01-21T08:09:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.226+0000 7f02b987d700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f02a002ff60 0x7f02a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.226+0000 7f02b987d700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f02a002ff60 0x7f02a0032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f02b00096a0 tx=0x7f02b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.230+0000 7f02aaffd700 1 -- 172.21.15.78:0/3163842338 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02a4014420 con 0x7f02b40f51a0 2024-01-21T08:09:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.436+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02b40fabe0 con 0x7f02b40f51a0 2024-01-21T08:09:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.437+0000 7f02aaffd700 1 -- 172.21.15.78:0/3163842338 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02a4010230 con 0x7f02b40f51a0 2024-01-21T08:09:52.439 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f02a002ff60 msgr2=0x7f02a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f02a002ff60 0x7f02a0032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f02b00096a0 tx=0x7f02b0006b40).stop 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 msgr2=0x7f02b40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40fd120 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f02a4013510 tx=0x7f02a400b040).stop 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 shutdown_connections 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f02a002ff60 0x7f02a0032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 --2- 172.21.15.78:0/3163842338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b40f51a0 0x7f02b40fd120 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 >> 172.21.15.78:0/3163842338 conn(0x7f02b40f09a0 msgr2=0x7f02b405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 shutdown_connections 2024-01-21T08:09:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:52.439+0000 7f02bb080700 1 -- 172.21.15.78:0/3163842338 wait complete. 2024-01-21T08:09:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:52.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:50.632889+0000 mgr.smithi046.lmxnwc (mgr.14164) 142 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:52.438116+0000 mon.smithi046 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.78:0/3163842338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:53.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:53.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:52.633205+0000 mgr.smithi046.lmxnwc (mgr.14164) 143 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.153+0000 7fc8f4bec700 1 -- 172.21.15.78:0/647354808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 msgr2=0x7fc8f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.153+0000 7fc8f4bec700 1 --2- 172.21.15.78:0/647354808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f00f6930 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fc8e4009fb0 tx=0x7fc8e4009b30).stop 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.153+0000 7fc8f4bec700 1 -- 172.21.15.78:0/647354808 shutdown_connections 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.153+0000 7fc8f4bec700 1 --2- 172.21.15.78:0/647354808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f00f6930 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.153+0000 7fc8f4bec700 1 -- 172.21.15.78:0/647354808 >> 172.21.15.78:0/647354808 conn(0x7fc8f00f1c70 msgr2=0x7fc8f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.153+0000 7fc8f4bec700 1 -- 172.21.15.78:0/647354808 shutdown_connections 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.153+0000 7fc8f4bec700 1 -- 172.21.15.78:0/647354808 wait complete. 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8f4bec700 1 Processor -- start 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8f4bec700 1 -- start start 2024-01-21T08:09:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8f4bec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8f4bec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0106c20 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8ee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8ee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49692/0 (socket says 172.21.15.78:49692) 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8ee59c700 1 -- 172.21.15.78:0/1588519306 learned_addr learned my addr 172.21.15.78:0/1588519306 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.154+0000 7fc8ee59c700 1 -- 172.21.15.78:0/1588519306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4005040 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.155+0000 7fc8ee59c700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f0106730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc8e4013510 tx=0x7fc8e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.155+0000 7fc8deffd700 1 -- 172.21.15.78:0/1588519306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e4009320 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.155+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f0106e20 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.155+0000 7fc8deffd700 1 -- 172.21.15.78:0/1588519306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8e4013f00 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.155+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f01072b0 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.155+0000 7fc8deffd700 1 -- 172.21.15.78:0/1588519306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e4014db0 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.156+0000 7fc8deffd700 1 -- 172.21.15.78:0/1588519306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8e401e420 con 0x7fc8f00f6510 2024-01-21T08:09:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.156+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f0045ad0 con 0x7fc8f00f6510 2024-01-21T08:09:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.156+0000 7fc8deffd700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc8d802ff60 0x7fc8d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.156+0000 7fc8deffd700 1 -- 172.21.15.78:0/1588519306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8e4041280 con 0x7fc8f00f6510 2024-01-21T08:09:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.156+0000 7fc8edd9b700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc8d802ff60 0x7fc8d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.157+0000 7fc8edd9b700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc8d802ff60 0x7fc8d8032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc8e00096a0 tx=0x7fc8e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.161+0000 7fc8deffd700 1 -- 172.21.15.78:0/1588519306 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8e4027770 con 0x7fc8f00f6510 2024-01-21T08:09:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.367+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8f00fbf60 con 0x7fc8f00f6510 2024-01-21T08:09:55.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.367+0000 7fc8deffd700 1 -- 172.21.15.78:0/1588519306 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc8e402b2d0 con 0x7fc8f00f6510 2024-01-21T08:09:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:55.369 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc8d802ff60 msgr2=0x7fc8d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc8d802ff60 0x7fc8d8032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc8e00096a0 tx=0x7fc8e0006b40).stop 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 msgr2=0x7fc8f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f0106730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc8e4013510 tx=0x7fc8e400b040).stop 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 shutdown_connections 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc8d802ff60 0x7fc8d8032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 --2- 172.21.15.78:0/1588519306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8f00f6510 0x7fc8f0106730 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 >> 172.21.15.78:0/1588519306 conn(0x7fc8f00f1c70 msgr2=0x7fc8f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:55.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 shutdown_connections 2024-01-21T08:09:55.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:55.369+0000 7fc8f4bec700 1 -- 172.21.15.78:0/1588519306 wait complete. 2024-01-21T08:09:55.372 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:54.633481+0000 mgr.smithi046.lmxnwc (mgr.14164) 144 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:55.368549+0000 mon.smithi046 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.78:0/1588519306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:56.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:56.799 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:09:57.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.968+0000 7f6abf9a3700 1 -- 172.21.15.78:0/4285126208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 msgr2=0x7f6ab80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.968+0000 7f6abf9a3700 1 --2- 172.21.15.78:0/4285126208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab80f90f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f6aa8009fb0 tx=0x7f6aa8009b30).stop 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.968+0000 7f6abf9a3700 1 -- 172.21.15.78:0/4285126208 shutdown_connections 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.968+0000 7f6abf9a3700 1 --2- 172.21.15.78:0/4285126208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab80f90f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.968+0000 7f6abf9a3700 1 -- 172.21.15.78:0/4285126208 >> 172.21.15.78:0/4285126208 conn(0x7f6ab80f0b70 msgr2=0x7f6ab80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.968+0000 7f6abf9a3700 1 -- 172.21.15.78:0/4285126208 shutdown_connections 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.968+0000 7f6abf9a3700 1 -- 172.21.15.78:0/4285126208 wait complete. 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abf9a3700 1 Processor -- start 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abf9a3700 1 -- start start 2024-01-21T08:09:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abf9a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abf9a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8106bd0 con 0x7f6ab8060280 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abd73f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abd73f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49710/0 (socket says 172.21.15.78:49710) 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abd73f700 1 -- 172.21.15.78:0/42519491 learned_addr learned my addr 172.21.15.78:0/42519491 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abd73f700 1 -- 172.21.15.78:0/42519491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa8005040 con 0x7f6ab8060280 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.969+0000 7f6abd73f700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab81066e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f6aa8013510 tx=0x7f6aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.970+0000 7f6aadffb700 1 -- 172.21.15.78:0/42519491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aa8009320 con 0x7f6ab8060280 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.970+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab8106dd0 con 0x7f6ab8060280 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.970+0000 7f6aadffb700 1 -- 172.21.15.78:0/42519491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6aa8013f00 con 0x7f6ab8060280 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.970+0000 7f6aadffb700 1 -- 172.21.15.78:0/42519491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aa8014dc0 con 0x7f6ab8060280 2024-01-21T08:09:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.970+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab8107260 con 0x7f6ab8060280 2024-01-21T08:09:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.971+0000 7f6aadffb700 1 -- 172.21.15.78:0/42519491 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6aa80095f0 con 0x7f6ab8060280 2024-01-21T08:09:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.971+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab8045ad0 con 0x7f6ab8060280 2024-01-21T08:09:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.971+0000 7f6aadffb700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6aa402ff60 0x7f6aa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:09:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.971+0000 7f6aadffb700 1 -- 172.21.15.78:0/42519491 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6aa8041020 con 0x7f6ab8060280 2024-01-21T08:09:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.971+0000 7f6abcf3e700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6aa402ff60 0x7f6aa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:09:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.972+0000 7f6abcf3e700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6aa402ff60 0x7f6aa4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f6ab40096a0 tx=0x7f6ab4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:09:57.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:57.976+0000 7f6aadffb700 1 -- 172.21.15.78:0/42519491 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6aa8014420 con 0x7f6ab8060280 2024-01-21T08:09:58.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.183+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ab8108110 con 0x7f6ab8060280 2024-01-21T08:09:58.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.184+0000 7f6aadffb700 1 -- 172.21.15.78:0/42519491 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6aa801c920 con 0x7f6ab8060280 2024-01-21T08:09:58.186 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:09:58.186 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6aa402ff60 msgr2=0x7f6aa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6aa402ff60 0x7f6aa4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f6ab40096a0 tx=0x7f6ab4006b40).stop 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 msgr2=0x7f6ab81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab81066e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f6aa8013510 tx=0x7f6aa800b040).stop 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 shutdown_connections 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f6aa402ff60 0x7f6aa4032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 --2- 172.21.15.78:0/42519491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ab8060280 0x7f6ab81066e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 >> 172.21.15.78:0/42519491 conn(0x7f6ab80f0b70 msgr2=0x7f6ab80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 shutdown_connections 2024-01-21T08:09:58.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:09:58.186+0000 7f6abf9a3700 1 -- 172.21.15.78:0/42519491 wait complete. 2024-01-21T08:09:58.189 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:09:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:56.633775+0000 mgr.smithi046.lmxnwc (mgr.14164) 145 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:09:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:09:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:09:58.185122+0000 mon.smithi046 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.78:0/42519491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:09:59.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:09:59.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:09:58.634057+0000 mgr.smithi046.lmxnwc (mgr.14164) 146 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:00.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.778+0000 7f792e343700 1 -- 172.21.15.78:0/955209667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 msgr2=0x7f79280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:00.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.778+0000 7f792e343700 1 --2- 172.21.15.78:0/955209667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f79280f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f7918009fb0 tx=0x7f7918009b30).stop 2024-01-21T08:10:00.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.778+0000 7f792e343700 1 -- 172.21.15.78:0/955209667 shutdown_connections 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.778+0000 7f792e343700 1 --2- 172.21.15.78:0/955209667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f79280f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.778+0000 7f792e343700 1 -- 172.21.15.78:0/955209667 >> 172.21.15.78:0/955209667 conn(0x7f79280f0b70 msgr2=0x7f79280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.778+0000 7f792e343700 1 -- 172.21.15.78:0/955209667 shutdown_connections 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.778+0000 7f792e343700 1 -- 172.21.15.78:0/955209667 wait complete. 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.779+0000 7f792e343700 1 Processor -- start 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.779+0000 7f792e343700 1 -- start start 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.779+0000 7f792e343700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f7928109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.779+0000 7f792e343700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7928060240 con 0x7f79280f5410 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7927fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f7928109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7927fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f7928109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:49738/0 (socket says 172.21.15.78:49738) 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7927fff700 1 -- 172.21.15.78:0/2951665193 learned_addr learned my addr 172.21.15.78:0/2951665193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:00.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7927fff700 1 -- 172.21.15.78:0/2951665193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918005040 con 0x7f79280f5410 2024-01-21T08:10:00.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7927fff700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f7928109430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7918013510 tx=0x7f791800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:00.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7924ff9700 1 -- 172.21.15.78:0/2951665193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7918009320 con 0x7f79280f5410 2024-01-21T08:10:00.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7924ff9700 1 -- 172.21.15.78:0/2951665193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7918013f00 con 0x7f79280f5410 2024-01-21T08:10:00.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7928060440 con 0x7f79280f5410 2024-01-21T08:10:00.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f7924ff9700 1 -- 172.21.15.78:0/2951665193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7918014d90 con 0x7f79280f5410 2024-01-21T08:10:00.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.780+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7928109ae0 con 0x7f79280f5410 2024-01-21T08:10:00.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.781+0000 7f7924ff9700 1 -- 172.21.15.78:0/2951665193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79180095f0 con 0x7f79280f5410 2024-01-21T08:10:00.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.781+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7928045ad0 con 0x7f79280f5410 2024-01-21T08:10:00.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.781+0000 7f7924ff9700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f791402ff60 0x7f7914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:00.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.781+0000 7f7924ff9700 1 -- 172.21.15.78:0/2951665193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7918040fe0 con 0x7f79280f5410 2024-01-21T08:10:00.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.782+0000 7f79277fe700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f791402ff60 0x7f7914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:00.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.782+0000 7f79277fe700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f791402ff60 0x7f7914032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f79100096a0 tx=0x7f7910006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:00.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.786+0000 7f7924ff9700 1 -- 172.21.15.78:0/2951665193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f791801d7f0 con 0x7f79280f5410 2024-01-21T08:10:00.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.992+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79280fae60 con 0x7f79280f5410 2024-01-21T08:10:00.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.993+0000 7f7924ff9700 1 -- 172.21.15.78:0/2951665193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7918010230 con 0x7f79280f5410 2024-01-21T08:10:00.996 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:00.996 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.994+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f791402ff60 msgr2=0x7f7914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.994+0000 7f792e343700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f791402ff60 0x7f7914032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f79100096a0 tx=0x7f7910006b40).stop 2024-01-21T08:10:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.994+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 msgr2=0x7f7928109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.994+0000 7f792e343700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f7928109430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7918013510 tx=0x7f791800b040).stop 2024-01-21T08:10:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.995+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 shutdown_connections 2024-01-21T08:10:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.995+0000 7f792e343700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f791402ff60 0x7f7914032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.995+0000 7f792e343700 1 --2- 172.21.15.78:0/2951665193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79280f5410 0x7f7928109430 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.995+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 >> 172.21.15.78:0/2951665193 conn(0x7f79280f0b70 msgr2=0x7f79280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.995+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 shutdown_connections 2024-01-21T08:10:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:00.995+0000 7f792e343700 1 -- 172.21.15.78:0/2951665193 wait complete. 2024-01-21T08:10:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:00.993932+0000 mon.smithi046 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.78:0/2951665193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:02.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:02.395 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:00.634425+0000 mgr.smithi046.lmxnwc (mgr.14164) 147 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:03.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.647+0000 7f833a0f9700 1 -- 172.21.15.78:0/2013243994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 msgr2=0x7f83340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:03.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.647+0000 7f833a0f9700 1 --2- 172.21.15.78:0/2013243994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f83340f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f8328009fb0 tx=0x7f8328009b30).stop 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 -- 172.21.15.78:0/2013243994 shutdown_connections 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 --2- 172.21.15.78:0/2013243994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f83340f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 -- 172.21.15.78:0/2013243994 >> 172.21.15.78:0/2013243994 conn(0x7f83340f1c70 msgr2=0x7f83340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 -- 172.21.15.78:0/2013243994 shutdown_connections 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 -- 172.21.15.78:0/2013243994 wait complete. 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 Processor -- start 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 -- start start 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f8334106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.648+0000 7f833a0f9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8334106c20 con 0x7f83340f6510 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f83337fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f8334106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f83337fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f8334106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:55074/0 (socket says 172.21.15.78:55074) 2024-01-21T08:10:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f83337fe700 1 -- 172.21.15.78:0/450871954 learned_addr learned my addr 172.21.15.78:0/450871954 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f83337fe700 1 -- 172.21.15.78:0/450871954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328005040 con 0x7f83340f6510 2024-01-21T08:10:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f83337fe700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f8334106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8328014000 tx=0x7f832800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f831bfff700 1 -- 172.21.15.78:0/450871954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83280095f0 con 0x7f83340f6510 2024-01-21T08:10:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8334106e20 con 0x7f83340f6510 2024-01-21T08:10:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f831bfff700 1 -- 172.21.15.78:0/450871954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8328007e70 con 0x7f83340f6510 2024-01-21T08:10:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f831bfff700 1 -- 172.21.15.78:0/450871954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8328014ea0 con 0x7f83340f6510 2024-01-21T08:10:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.649+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83341072b0 con 0x7f83340f6510 2024-01-21T08:10:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.650+0000 7f831bfff700 1 -- 172.21.15.78:0/450871954 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8328025070 con 0x7f83340f6510 2024-01-21T08:10:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.650+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8334045ad0 con 0x7f83340f6510 2024-01-21T08:10:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.650+0000 7f831bfff700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f831c02ff60 0x7f831c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.650+0000 7f831bfff700 1 -- 172.21.15.78:0/450871954 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8328040a10 con 0x7f83340f6510 2024-01-21T08:10:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.651+0000 7f8332ffd700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f831c02ff60 0x7f831c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.651+0000 7f8332ffd700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f831c02ff60 0x7f831c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f83240096a0 tx=0x7f8324006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.655+0000 7f831bfff700 1 -- 172.21.15.78:0/450871954 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f832800f4e0 con 0x7f83340f6510 2024-01-21T08:10:03.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.861+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83340fbf60 con 0x7f83340f6510 2024-01-21T08:10:03.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.862+0000 7f831bfff700 1 -- 172.21.15.78:0/450871954 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f832800f300 con 0x7f83340f6510 2024-01-21T08:10:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:03.864 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f831c02ff60 msgr2=0x7f831c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f831c02ff60 0x7f831c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f83240096a0 tx=0x7f8324006b40).stop 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 msgr2=0x7f8334106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f8334106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8328014000 tx=0x7f832800b040).stop 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 shutdown_connections 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f831c02ff60 0x7f831c032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 --2- 172.21.15.78:0/450871954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83340f6510 0x7f8334106730 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 >> 172.21.15.78:0/450871954 conn(0x7f83340f1c70 msgr2=0x7f83340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 shutdown_connections 2024-01-21T08:10:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:03.863+0000 7f833a0f9700 1 -- 172.21.15.78:0/450871954 wait complete. 2024-01-21T08:10:03.866 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:04.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:02.634835+0000 mgr.smithi046.lmxnwc (mgr.14164) 148 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:04.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:03.862753+0000 mon.smithi046 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.78:0/450871954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:05.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:05.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 -- 172.21.15.78:0/2411117282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 msgr2=0x7f68780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 --2- 172.21.15.78:0/2411117282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f68780f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f6860009fb0 tx=0x7f6860009b30).stop 2024-01-21T08:10:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 -- 172.21.15.78:0/2411117282 shutdown_connections 2024-01-21T08:10:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 --2- 172.21.15.78:0/2411117282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f68780f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 -- 172.21.15.78:0/2411117282 >> 172.21.15.78:0/2411117282 conn(0x7f68780f1c70 msgr2=0x7f68780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 -- 172.21.15.78:0/2411117282 shutdown_connections 2024-01-21T08:10:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 -- 172.21.15.78:0/2411117282 wait complete. 2024-01-21T08:10:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.450+0000 7f687cb03700 1 Processor -- start 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687cb03700 1 -- start start 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687cb03700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f6878106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687cb03700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6878106c20 con 0x7f68780f6510 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f6878106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f6878106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:55092/0 (socket says 172.21.15.78:55092) 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687659c700 1 -- 172.21.15.78:0/2442007342 learned_addr learned my addr 172.21.15.78:0/2442007342 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687659c700 1 -- 172.21.15.78:0/2442007342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6860005040 con 0x7f68780f6510 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.451+0000 7f687659c700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f6878106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f6860013510 tx=0x7f686000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.452+0000 7f686effd700 1 -- 172.21.15.78:0/2442007342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6860009320 con 0x7f68780f6510 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.452+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6878106e20 con 0x7f68780f6510 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.452+0000 7f686effd700 1 -- 172.21.15.78:0/2442007342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6860013f00 con 0x7f68780f6510 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.452+0000 7f686effd700 1 -- 172.21.15.78:0/2442007342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6860014d90 con 0x7f68780f6510 2024-01-21T08:10:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.452+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68781072b0 con 0x7f68780f6510 2024-01-21T08:10:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.453+0000 7f686effd700 1 -- 172.21.15.78:0/2442007342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68600095f0 con 0x7f68780f6510 2024-01-21T08:10:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.453+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6878045ad0 con 0x7f68780f6510 2024-01-21T08:10:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.453+0000 7f686effd700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f686402ff60 0x7f6864032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.453+0000 7f686effd700 1 -- 172.21.15.78:0/2442007342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6860041000 con 0x7f68780f6510 2024-01-21T08:10:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.453+0000 7f6875d9b700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f686402ff60 0x7f6864032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.453+0000 7f6875d9b700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f686402ff60 0x7f6864032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f68680096a0 tx=0x7f6868006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.458+0000 7f686effd700 1 -- 172.21.15.78:0/2442007342 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6860014420 con 0x7f68780f6510 2024-01-21T08:10:06.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.664+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68780fbf60 con 0x7f68780f6510 2024-01-21T08:10:06.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.665+0000 7f686effd700 1 -- 172.21.15.78:0/2442007342 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f686001d7f0 con 0x7f68780f6510 2024-01-21T08:10:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:06.667 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:06.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f686402ff60 msgr2=0x7f6864032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f686402ff60 0x7f6864032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f68680096a0 tx=0x7f6868006b40).stop 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 msgr2=0x7f6878106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f6878106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f6860013510 tx=0x7f686000b040).stop 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 shutdown_connections 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f686402ff60 0x7f6864032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 --2- 172.21.15.78:0/2442007342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68780f6510 0x7f6878106730 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 >> 172.21.15.78:0/2442007342 conn(0x7f68780f1c70 msgr2=0x7f68780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 shutdown_connections 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:06.667+0000 7f687cb03700 1 -- 172.21.15.78:0/2442007342 wait complete. 2024-01-21T08:10:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:04.635237+0000 mgr.smithi046.lmxnwc (mgr.14164) 149 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:06.665954+0000 mon.smithi046 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.78:0/2442007342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:08.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:08.094 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:06.635637+0000 mgr.smithi046.lmxnwc (mgr.14164) 150 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:09.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.276+0000 7f39b89e8700 1 -- 172.21.15.78:0/1076219346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 msgr2=0x7f39b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:09.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.276+0000 7f39b89e8700 1 --2- 172.21.15.78:0/1076219346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b40f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f39a4009960 tx=0x7f39a4004d10).stop 2024-01-21T08:10:09.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.276+0000 7f39b89e8700 1 -- 172.21.15.78:0/1076219346 shutdown_connections 2024-01-21T08:10:09.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.276+0000 7f39b89e8700 1 --2- 172.21.15.78:0/1076219346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b40f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:09.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.276+0000 7f39b89e8700 1 -- 172.21.15.78:0/1076219346 >> 172.21.15.78:0/1076219346 conn(0x7f39b40f1c70 msgr2=0x7f39b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:09.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.276+0000 7f39b89e8700 1 -- 172.21.15.78:0/1076219346 shutdown_connections 2024-01-21T08:10:09.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.276+0000 7f39b89e8700 1 -- 172.21.15.78:0/1076219346 wait complete. 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b89e8700 1 Processor -- start 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b89e8700 1 -- start start 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b89e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b89e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b4106bc0 con 0x7f39b40f6510 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:55102/0 (socket says 172.21.15.78:55102) 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b259c700 1 -- 172.21.15.78:0/280901003 learned_addr learned my addr 172.21.15.78:0/280901003 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:09.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.277+0000 7f39b259c700 1 -- 172.21.15.78:0/280901003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a4009640 con 0x7f39b40f6510 2024-01-21T08:10:09.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.278+0000 7f39b259c700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b41066d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f39a4004b70 tx=0x7f39a4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:09.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.278+0000 7f39aaffd700 1 -- 172.21.15.78:0/280901003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39a4018070 con 0x7f39b40f6510 2024-01-21T08:10:09.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.278+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b4106dc0 con 0x7f39b40f6510 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.278+0000 7f39aaffd700 1 -- 172.21.15.78:0/280901003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39a400b870 con 0x7f39b40f6510 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.278+0000 7f39aaffd700 1 -- 172.21.15.78:0/280901003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39a401c560 con 0x7f39b40f6510 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.278+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b4107250 con 0x7f39b40f6510 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.279+0000 7f39aaffd700 1 -- 172.21.15.78:0/280901003 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39a401c6c0 con 0x7f39b40f6510 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.279+0000 7f39aaffd700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f39a002ff60 0x7f39a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.279+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b4045ad0 con 0x7f39b40f6510 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.279+0000 7f39aaffd700 1 -- 172.21.15.78:0/280901003 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39a40411f0 con 0x7f39b40f6510 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.279+0000 7f39b1d9b700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f39a002ff60 0x7f39a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:09.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.279+0000 7f39b1d9b700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f39a002ff60 0x7f39a0032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f399c0096a0 tx=0x7f399c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:09.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.284+0000 7f39aaffd700 1 -- 172.21.15.78:0/280901003 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39a4012c00 con 0x7f39b40f6510 2024-01-21T08:10:09.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.489+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39b4108100 con 0x7f39b40f6510 2024-01-21T08:10:09.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.490+0000 7f39aaffd700 1 -- 172.21.15.78:0/280901003 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f39a40127f0 con 0x7f39b40f6510 2024-01-21T08:10:09.492 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:09.492 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f39a002ff60 msgr2=0x7f39a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f39a002ff60 0x7f39a0032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f399c0096a0 tx=0x7f399c006b40).stop 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 msgr2=0x7f39b41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b41066d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f39a4004b70 tx=0x7f39a4004d10).stop 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 shutdown_connections 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f39a002ff60 0x7f39a0032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 --2- 172.21.15.78:0/280901003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39b40f6510 0x7f39b41066d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.491+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 >> 172.21.15.78:0/280901003 conn(0x7f39b40f1c70 msgr2=0x7f39b40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.492+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 shutdown_connections 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:09.492+0000 7f39b89e8700 1 -- 172.21.15.78:0/280901003 wait complete. 2024-01-21T08:10:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:08.635984+0000 mgr.smithi046.lmxnwc (mgr.14164) 151 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:09.490928+0000 mon.smithi046 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.78:0/280901003' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:10.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:10.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.053+0000 7f4e4b565700 1 -- 172.21.15.78:0/2940006929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 msgr2=0x7f4e440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.053+0000 7f4e4b565700 1 --2- 172.21.15.78:0/2940006929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e440f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f4e34009fb0 tx=0x7f4e34009b30).stop 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.053+0000 7f4e4b565700 1 -- 172.21.15.78:0/2940006929 shutdown_connections 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.053+0000 7f4e4b565700 1 --2- 172.21.15.78:0/2940006929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e440f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.053+0000 7f4e4b565700 1 -- 172.21.15.78:0/2940006929 >> 172.21.15.78:0/2940006929 conn(0x7f4e440f09a0 msgr2=0x7f4e440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.053+0000 7f4e4b565700 1 -- 172.21.15.78:0/2940006929 shutdown_connections 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.053+0000 7f4e4b565700 1 -- 172.21.15.78:0/2940006929 wait complete. 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.054+0000 7f4e4b565700 1 Processor -- start 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.054+0000 7f4e4b565700 1 -- start start 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.054+0000 7f4e4b565700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e4410cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.054+0000 7f4e4b565700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4410cfb0 con 0x7f4e440f51a0 2024-01-21T08:10:12.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.054+0000 7f4e4a563700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e4410cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.054+0000 7f4e4a563700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e4410cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:55124/0 (socket says 172.21.15.78:55124) 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.054+0000 7f4e4a563700 1 -- 172.21.15.78:0/1188498893 learned_addr learned my addr 172.21.15.78:0/1188498893 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.055+0000 7f4e4a563700 1 -- 172.21.15.78:0/1188498893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e34005040 con 0x7f4e440f51a0 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.055+0000 7f4e4a563700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e4410cac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4e34013510 tx=0x7f4e3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.055+0000 7f4e3affd700 1 -- 172.21.15.78:0/1188498893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e34009320 con 0x7f4e440f51a0 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.055+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4410d1b0 con 0x7f4e440f51a0 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.055+0000 7f4e3affd700 1 -- 172.21.15.78:0/1188498893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e34013f00 con 0x7f4e440f51a0 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.055+0000 7f4e3affd700 1 -- 172.21.15.78:0/1188498893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e34014dc0 con 0x7f4e440f51a0 2024-01-21T08:10:12.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.055+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4410d640 con 0x7f4e440f51a0 2024-01-21T08:10:12.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.056+0000 7f4e3affd700 1 -- 172.21.15.78:0/1188498893 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e34014420 con 0x7f4e440f51a0 2024-01-21T08:10:12.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.056+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e44045ad0 con 0x7f4e440f51a0 2024-01-21T08:10:12.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.056+0000 7f4e3affd700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e3002ff60 0x7f4e30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:12.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.056+0000 7f4e3affd700 1 -- 172.21.15.78:0/1188498893 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e340410e0 con 0x7f4e440f51a0 2024-01-21T08:10:12.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.056+0000 7f4e49d62700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e3002ff60 0x7f4e30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:12.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.057+0000 7f4e49d62700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e3002ff60 0x7f4e30032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4e400096a0 tx=0x7f4e40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:12.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.061+0000 7f4e3affd700 1 -- 172.21.15.78:0/1188498893 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e340095f0 con 0x7f4e440f51a0 2024-01-21T08:10:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.267+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e4410e4f0 con 0x7f4e440f51a0 2024-01-21T08:10:12.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.268+0000 7f4e3affd700 1 -- 172.21.15.78:0/1188498893 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e3401cb10 con 0x7f4e440f51a0 2024-01-21T08:10:12.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:12.270 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e3002ff60 msgr2=0x7f4e30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e3002ff60 0x7f4e30032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4e400096a0 tx=0x7f4e40006b40).stop 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 msgr2=0x7f4e4410cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e4410cac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4e34013510 tx=0x7f4e3400b040).stop 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 shutdown_connections 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e3002ff60 0x7f4e30032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 --2- 172.21.15.78:0/1188498893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440f51a0 0x7f4e4410cac0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 >> 172.21.15.78:0/1188498893 conn(0x7f4e440f09a0 msgr2=0x7f4e440f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:12.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 shutdown_connections 2024-01-21T08:10:12.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:12.270+0000 7f4e4b565700 1 -- 172.21.15.78:0/1188498893 wait complete. 2024-01-21T08:10:12.273 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:10.636330+0000 mgr.smithi046.lmxnwc (mgr.14164) 152 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:12.268997+0000 mon.smithi046 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.78:0/1188498893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:13.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:13.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:14.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:12.636786+0000 mgr.smithi046.lmxnwc (mgr.14164) 153 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:14.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 -- 172.21.15.78:0/4025048777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 msgr2=0x7f0c6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:14.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 --2- 172.21.15.78:0/4025048777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c0f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f0c60009fb0 tx=0x7f0c60009b30).stop 2024-01-21T08:10:14.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 -- 172.21.15.78:0/4025048777 shutdown_connections 2024-01-21T08:10:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 --2- 172.21.15.78:0/4025048777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c0f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 -- 172.21.15.78:0/4025048777 >> 172.21.15.78:0/4025048777 conn(0x7f0c6c0f1c70 msgr2=0x7f0c6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 -- 172.21.15.78:0/4025048777 shutdown_connections 2024-01-21T08:10:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 -- 172.21.15.78:0/4025048777 wait complete. 2024-01-21T08:10:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 Processor -- start 2024-01-21T08:10:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.893+0000 7f0c72b45700 1 -- start start 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.894+0000 7f0c72b45700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.894+0000 7f0c72b45700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c106c20 con 0x7f0c6c0f6510 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.894+0000 7f0c708e1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.894+0000 7f0c708e1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:47950/0 (socket says 172.21.15.78:47950) 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.894+0000 7f0c708e1700 1 -- 172.21.15.78:0/1379005996 learned_addr learned my addr 172.21.15.78:0/1379005996 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.894+0000 7f0c708e1700 1 -- 172.21.15.78:0/1379005996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c60005040 con 0x7f0c6c0f6510 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.894+0000 7f0c708e1700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f0c60013510 tx=0x7f0c6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c697fa700 1 -- 172.21.15.78:0/1379005996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c60009320 con 0x7f0c6c0f6510 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c106e20 con 0x7f0c6c0f6510 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c697fa700 1 -- 172.21.15.78:0/1379005996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c60013f00 con 0x7f0c6c0f6510 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c697fa700 1 -- 172.21.15.78:0/1379005996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c60014d90 con 0x7f0c6c0f6510 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c1072b0 con 0x7f0c6c0f6510 2024-01-21T08:10:14.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c697fa700 1 -- 172.21.15.78:0/1379005996 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c600263a0 con 0x7f0c6c0f6510 2024-01-21T08:10:14.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c045ad0 con 0x7f0c6c0f6510 2024-01-21T08:10:14.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.895+0000 7f0c697fa700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0c5c02ff60 0x7f0c5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:14.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.896+0000 7f0c697fa700 1 -- 172.21.15.78:0/1379005996 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c60041300 con 0x7f0c6c0f6510 2024-01-21T08:10:14.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.896+0000 7f0c6bfff700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0c5c02ff60 0x7f0c5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:14.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.896+0000 7f0c6bfff700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0c5c02ff60 0x7f0c5c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0c580096a0 tx=0x7f0c58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:14.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:14.901+0000 7f0c697fa700 1 -- 172.21.15.78:0/1379005996 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c600126f0 con 0x7f0c6c0f6510 2024-01-21T08:10:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.106+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c6c0fc050 con 0x7f0c6c0f6510 2024-01-21T08:10:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.107+0000 7f0c697fa700 1 -- 172.21.15.78:0/1379005996 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c6000f340 con 0x7f0c6c0f6510 2024-01-21T08:10:15.109 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:15.109 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.108+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0c5c02ff60 msgr2=0x7f0c5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.108+0000 7f0c72b45700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0c5c02ff60 0x7f0c5c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0c580096a0 tx=0x7f0c58006b40).stop 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.108+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 msgr2=0x7f0c6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.108+0000 7f0c72b45700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f0c60013510 tx=0x7f0c6000b040).stop 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.109+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 shutdown_connections 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.109+0000 7f0c72b45700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0c5c02ff60 0x7f0c5c032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.109+0000 7f0c72b45700 1 --2- 172.21.15.78:0/1379005996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c106730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.109+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 >> 172.21.15.78:0/1379005996 conn(0x7f0c6c0f1c70 msgr2=0x7f0c6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.109+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 shutdown_connections 2024-01-21T08:10:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:15.109+0000 7f0c72b45700 1 -- 172.21.15.78:0/1379005996 wait complete. 2024-01-21T08:10:15.112 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:15.108038+0000 mon.smithi046 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.78:0/1379005996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:16.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:16.527 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:14.637262+0000 mgr.smithi046.lmxnwc (mgr.14164) 154 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.739+0000 7f90b0f57700 1 -- 172.21.15.78:0/3159672277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 msgr2=0x7f90ac0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.739+0000 7f90b0f57700 1 --2- 172.21.15.78:0/3159672277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac0f9150 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f9094009fb0 tx=0x7f9094009b30).stop 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.739+0000 7f90b0f57700 1 -- 172.21.15.78:0/3159672277 shutdown_connections 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.739+0000 7f90b0f57700 1 --2- 172.21.15.78:0/3159672277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac0f9150 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.739+0000 7f90b0f57700 1 -- 172.21.15.78:0/3159672277 >> 172.21.15.78:0/3159672277 conn(0x7f90ac0f0b70 msgr2=0x7f90ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.739+0000 7f90b0f57700 1 -- 172.21.15.78:0/3159672277 shutdown_connections 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.739+0000 7f90b0f57700 1 -- 172.21.15.78:0/3159672277 wait complete. 2024-01-21T08:10:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.740+0000 7f90b0f57700 1 Processor -- start 2024-01-21T08:10:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.740+0000 7f90b0f57700 1 -- start start 2024-01-21T08:10:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.740+0000 7f90b0f57700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.740+0000 7f90b0f57700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac106bd0 con 0x7f90ac060220 2024-01-21T08:10:17.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.740+0000 7f90aa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:17.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.740+0000 7f90aa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:47964/0 (socket says 172.21.15.78:47964) 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90aa59c700 1 -- 172.21.15.78:0/3969530541 learned_addr learned my addr 172.21.15.78:0/3969530541 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90aa59c700 1 -- 172.21.15.78:0/3969530541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9094005040 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90aa59c700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac1066e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f9094013510 tx=0x7f909400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90a2ffd700 1 -- 172.21.15.78:0/3969530541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9094009320 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90a2ffd700 1 -- 172.21.15.78:0/3969530541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9094013f00 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90a2ffd700 1 -- 172.21.15.78:0/3969530541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9094014dc0 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac106dd0 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.741+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac107260 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.742+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ac045ad0 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.742+0000 7f90a2ffd700 1 -- 172.21.15.78:0/3969530541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9094014420 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.742+0000 7f90a2ffd700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f909802ff60 0x7f9098032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.742+0000 7f90a2ffd700 1 -- 172.21.15.78:0/3969530541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9094041030 con 0x7f90ac060220 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.742+0000 7f90a9d9b700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f909802ff60 0x7f9098032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.743+0000 7f90a9d9b700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f909802ff60 0x7f9098032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f909c0096a0 tx=0x7f909c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:17.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.747+0000 7f90a2ffd700 1 -- 172.21.15.78:0/3969530541 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90940095f0 con 0x7f90ac060220 2024-01-21T08:10:17.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.950+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90ac108110 con 0x7f90ac060220 2024-01-21T08:10:17.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.951+0000 7f90a2ffd700 1 -- 172.21.15.78:0/3969530541 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f909401cb10 con 0x7f90ac060220 2024-01-21T08:10:17.953 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:17.953 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:17.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f909802ff60 msgr2=0x7f9098032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:17.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f909802ff60 0x7f9098032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f909c0096a0 tx=0x7f909c006b40).stop 2024-01-21T08:10:17.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 msgr2=0x7f90ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:17.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac1066e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f9094013510 tx=0x7f909400b040).stop 2024-01-21T08:10:17.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 shutdown_connections 2024-01-21T08:10:17.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f909802ff60 0x7f9098032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:17.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 --2- 172.21.15.78:0/3969530541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac060220 0x7f90ac1066e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:17.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 >> 172.21.15.78:0/3969530541 conn(0x7f90ac0f0b70 msgr2=0x7f90ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:17.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 shutdown_connections 2024-01-21T08:10:17.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:17.952+0000 7f90b0f57700 1 -- 172.21.15.78:0/3969530541 wait complete. 2024-01-21T08:10:17.955 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:18.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:16.637567+0000 mgr.smithi046.lmxnwc (mgr.14164) 155 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:18.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:17.951958+0000 mon.smithi046 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.78:0/3969530541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:19.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:19.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:20.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.508+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2544227681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 msgr2=0x7fa19c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.508+0000 7fa1a2c91700 1 --2- 172.21.15.78:0/2544227681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c0f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fa190009fb0 tx=0x7fa190009b30).stop 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.508+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2544227681 shutdown_connections 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.508+0000 7fa1a2c91700 1 --2- 172.21.15.78:0/2544227681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c0f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.508+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2544227681 >> 172.21.15.78:0/2544227681 conn(0x7fa19c0f0b70 msgr2=0x7fa19c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.508+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2544227681 shutdown_connections 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.508+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2544227681 wait complete. 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.509+0000 7fa1a2c91700 1 Processor -- start 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.509+0000 7fa1a2c91700 1 -- start start 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.509+0000 7fa1a2c91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c1093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.509+0000 7fa1a2c91700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c060240 con 0x7fa19c0f5410 2024-01-21T08:10:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.509+0000 7fa1a0a2d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c1093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.509+0000 7fa1a0a2d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c1093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:47978/0 (socket says 172.21.15.78:47978) 2024-01-21T08:10:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.509+0000 7fa1a0a2d700 1 -- 172.21.15.78:0/2326742694 learned_addr learned my addr 172.21.15.78:0/2326742694 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.510+0000 7fa1a0a2d700 1 -- 172.21.15.78:0/2326742694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190005040 con 0x7fa19c0f5410 2024-01-21T08:10:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.510+0000 7fa1a0a2d700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c1093d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fa190014000 tx=0x7fa19000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.510+0000 7fa1997fa700 1 -- 172.21.15.78:0/2326742694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1900095f0 con 0x7fa19c0f5410 2024-01-21T08:10:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.510+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19c060440 con 0x7fa19c0f5410 2024-01-21T08:10:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.510+0000 7fa1997fa700 1 -- 172.21.15.78:0/2326742694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa190007e70 con 0x7fa19c0f5410 2024-01-21T08:10:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.510+0000 7fa1997fa700 1 -- 172.21.15.78:0/2326742694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa190014ea0 con 0x7fa19c0f5410 2024-01-21T08:10:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.510+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19c109a80 con 0x7fa19c0f5410 2024-01-21T08:10:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.511+0000 7fa1997fa700 1 -- 172.21.15.78:0/2326742694 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1900079c0 con 0x7fa19c0f5410 2024-01-21T08:10:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.511+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19c045ad0 con 0x7fa19c0f5410 2024-01-21T08:10:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.511+0000 7fa1997fa700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa18c02ff60 0x7fa18c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.511+0000 7fa1997fa700 1 -- 172.21.15.78:0/2326742694 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa190040920 con 0x7fa19c0f5410 2024-01-21T08:10:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.511+0000 7fa19bfff700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa18c02ff60 0x7fa18c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.512+0000 7fa19bfff700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa18c02ff60 0x7fa18c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa1880096a0 tx=0x7fa188006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:20.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.516+0000 7fa1997fa700 1 -- 172.21.15.78:0/2326742694 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa19000f950 con 0x7fa19c0f5410 2024-01-21T08:10:20.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.723+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa19c10a2a0 con 0x7fa19c0f5410 2024-01-21T08:10:20.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.724+0000 7fa1997fa700 1 -- 172.21.15.78:0/2326742694 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa190010230 con 0x7fa19c0f5410 2024-01-21T08:10:20.726 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:20.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa18c02ff60 msgr2=0x7fa18c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa18c02ff60 0x7fa18c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa1880096a0 tx=0x7fa188006b40).stop 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 msgr2=0x7fa19c1093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c1093d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fa190014000 tx=0x7fa19000b040).stop 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 shutdown_connections 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa18c02ff60 0x7fa18c032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 --2- 172.21.15.78:0/2326742694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa19c0f5410 0x7fa19c1093d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 >> 172.21.15.78:0/2326742694 conn(0x7fa19c0f0b70 msgr2=0x7fa19c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 shutdown_connections 2024-01-21T08:10:20.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:20.726+0000 7fa1a2c91700 1 -- 172.21.15.78:0/2326742694 wait complete. 2024-01-21T08:10:20.729 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:18.637895+0000 mgr.smithi046.lmxnwc (mgr.14164) 156 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:20.725061+0000 mon.smithi046 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.78:0/2326742694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:20.743214+0000 mon.smithi046 (mon.0) 232 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:10:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:20.747206+0000 mon.smithi046 (mon.0) 233 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:10:22.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:22.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:20.638226+0000 mgr.smithi046.lmxnwc (mgr.14164) 157 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:23.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.323+0000 7f3a76434700 1 -- 172.21.15.78:0/843952648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700f5410 msgr2=0x7f3a700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:23.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.323+0000 7f3a76434700 1 --2- 172.21.15.78:0/843952648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700f5410 0x7f3a700f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f3a60009fb0 tx=0x7f3a60009b30).stop 2024-01-21T08:10:23.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.323+0000 7f3a76434700 1 -- 172.21.15.78:0/843952648 shutdown_connections 2024-01-21T08:10:23.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.323+0000 7f3a76434700 1 --2- 172.21.15.78:0/843952648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700f5410 0x7f3a700f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:23.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.323+0000 7f3a76434700 1 -- 172.21.15.78:0/843952648 >> 172.21.15.78:0/843952648 conn(0x7f3a700f0b70 msgr2=0x7f3a700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.323+0000 7f3a76434700 1 -- 172.21.15.78:0/843952648 shutdown_connections 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.323+0000 7f3a76434700 1 -- 172.21.15.78:0/843952648 wait complete. 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.324+0000 7f3a76434700 1 Processor -- start 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.324+0000 7f3a76434700 1 -- start start 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.324+0000 7f3a76434700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700601e0 0x7f3a70109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.324+0000 7f3a76434700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70109a20 con 0x7f3a700601e0 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.324+0000 7f3a6ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700601e0 0x7f3a70109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a6ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700601e0 0x7f3a70109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52444/0 (socket says 172.21.15.78:52444) 2024-01-21T08:10:23.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a6ffff700 1 -- 172.21.15.78:0/2047278293 learned_addr learned my addr 172.21.15.78:0/2047278293 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a6ffff700 1 -- 172.21.15.78:0/2047278293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a60005040 con 0x7f3a700601e0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a6ffff700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700601e0 0x7f3a70109530 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f3a60009f80 tx=0x7f3a60009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a6d7fa700 1 -- 172.21.15.78:0/2047278293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a600136c0 con 0x7f3a700601e0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a70109b60 con 0x7f3a700601e0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a6d7fa700 1 -- 172.21.15.78:0/2047278293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a600095f0 con 0x7f3a700601e0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a6d7fa700 1 -- 172.21.15.78:0/2047278293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a60014420 con 0x7f3a700601e0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.325+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a70109f90 con 0x7f3a700601e0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.326+0000 7f3a6d7fa700 1 -- 172.21.15.78:0/2047278293 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a60025060 con 0x7f3a700601e0 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.326+0000 7f3a6d7fa700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3a5c02ff60 0x7f3a5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.326+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a70059100 con 0x7f3a700601e0 2024-01-21T08:10:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.326+0000 7f3a6d7fa700 1 -- 172.21.15.78:0/2047278293 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a60040e00 con 0x7f3a700601e0 2024-01-21T08:10:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.326+0000 7f3a67fff700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3a5c02ff60 0x7f3a5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.326+0000 7f3a67fff700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3a5c02ff60 0x7f3a5c032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f3a580096a0 tx=0x7f3a58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:23.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.332+0000 7f3a6d7fa700 1 -- 172.21.15.78:0/2047278293 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a60028440 con 0x7f3a700601e0 2024-01-21T08:10:23.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.537+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a70045ad0 con 0x7f3a700601e0 2024-01-21T08:10:23.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.537+0000 7f3a6d7fa700 1 -- 172.21.15.78:0/2047278293 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a60026340 con 0x7f3a700601e0 2024-01-21T08:10:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:23.540 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3a5c02ff60 msgr2=0x7f3a5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3a5c02ff60 0x7f3a5c032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f3a580096a0 tx=0x7f3a58006b40).stop 2024-01-21T08:10:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700601e0 msgr2=0x7f3a70109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700601e0 0x7f3a70109530 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f3a60009f80 tx=0x7f3a60009b30).stop 2024-01-21T08:10:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 shutdown_connections 2024-01-21T08:10:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3a5c02ff60 0x7f3a5c032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 --2- 172.21.15.78:0/2047278293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a700601e0 0x7f3a70109530 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 >> 172.21.15.78:0/2047278293 conn(0x7f3a700f0b70 msgr2=0x7f3a700f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 shutdown_connections 2024-01-21T08:10:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:23.539+0000 7f3a76434700 1 -- 172.21.15.78:0/2047278293 wait complete. 2024-01-21T08:10:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:22.638582+0000 mgr.smithi046.lmxnwc (mgr.14164) 158 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:23.538610+0000 mon.smithi046 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.78:0/2047278293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:24.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:24.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:24.638930+0000 mgr.smithi046.lmxnwc (mgr.14164) 159 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 -- 172.21.15.78:0/1558429317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 msgr2=0x7fd53c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 --2- 172.21.15.78:0/1558429317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c0f7d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fd530009fb0 tx=0x7fd530009b30).stop 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 -- 172.21.15.78:0/1558429317 shutdown_connections 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 --2- 172.21.15.78:0/1558429317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c0f7d00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 -- 172.21.15.78:0/1558429317 >> 172.21.15.78:0/1558429317 conn(0x7fd53c0f0120 msgr2=0x7fd53c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 -- 172.21.15.78:0/1558429317 shutdown_connections 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 -- 172.21.15.78:0/1558429317 wait complete. 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 Processor -- start 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 -- start start 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.036+0000 7fd541e39700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1069d0 con 0x7fd53c0f78e0 2024-01-21T08:10:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd540e37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd540e37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52448/0 (socket says 172.21.15.78:52448) 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd540e37700 1 -- 172.21.15.78:0/1982459435 learned_addr learned my addr 172.21.15.78:0/1982459435 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd540e37700 1 -- 172.21.15.78:0/1982459435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530005040 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd540e37700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c1064e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fd530013510 tx=0x7fd53000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd5397fa700 1 -- 172.21.15.78:0/1982459435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd530009320 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c106bd0 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd5397fa700 1 -- 172.21.15.78:0/1982459435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd530013f00 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd5397fa700 1 -- 172.21.15.78:0/1982459435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd530014dc0 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.037+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c107060 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.038+0000 7fd5397fa700 1 -- 172.21.15.78:0/1982459435 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd530014420 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.038+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c045ad0 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.038+0000 7fd5397fa700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd52c02ff60 0x7fd52c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.038+0000 7fd5397fa700 1 -- 172.21.15.78:0/1982459435 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd530041110 con 0x7fd53c0f78e0 2024-01-21T08:10:26.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.038+0000 7fd53bfff700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd52c02ff60 0x7fd52c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:26.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.039+0000 7fd53bfff700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd52c02ff60 0x7fd52c032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd5280096a0 tx=0x7fd528006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:26.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.043+0000 7fd5397fa700 1 -- 172.21.15.78:0/1982459435 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5300095f0 con 0x7fd53c0f78e0 2024-01-21T08:10:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.252+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd53c107f10 con 0x7fd53c0f78e0 2024-01-21T08:10:26.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.253+0000 7fd5397fa700 1 -- 172.21.15.78:0/1982459435 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd53001cb10 con 0x7fd53c0f78e0 2024-01-21T08:10:26.255 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:26.255 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:26.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.254+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd52c02ff60 msgr2=0x7fd52c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:26.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.254+0000 7fd541e39700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd52c02ff60 0x7fd52c032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd5280096a0 tx=0x7fd528006b40).stop 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.254+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 msgr2=0x7fd53c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.254+0000 7fd541e39700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c1064e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fd530013510 tx=0x7fd53000b040).stop 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.254+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 shutdown_connections 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.255+0000 7fd541e39700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd52c02ff60 0x7fd52c032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.255+0000 7fd541e39700 1 --2- 172.21.15.78:0/1982459435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd53c0f78e0 0x7fd53c1064e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.255+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 >> 172.21.15.78:0/1982459435 conn(0x7fd53c0f0120 msgr2=0x7fd53c0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.255+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 shutdown_connections 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:26.255+0000 7fd541e39700 1 -- 172.21.15.78:0/1982459435 wait complete. 2024-01-21T08:10:26.257 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:26.253939+0000 mon.smithi046 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.78:0/1982459435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:27.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:27.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:26.639171+0000 mgr.smithi046.lmxnwc (mgr.14164) 160 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:28.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.765+0000 7fc40247e700 1 -- 172.21.15.78:0/819612127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 msgr2=0x7fc3fc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:28.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.765+0000 7fc40247e700 1 --2- 172.21.15.78:0/819612127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc0f90f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fc3ec009fb0 tx=0x7fc3ec009b30).stop 2024-01-21T08:10:28.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.765+0000 7fc40247e700 1 -- 172.21.15.78:0/819612127 shutdown_connections 2024-01-21T08:10:28.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.765+0000 7fc40247e700 1 --2- 172.21.15.78:0/819612127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc0f90f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:28.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.765+0000 7fc40247e700 1 -- 172.21.15.78:0/819612127 >> 172.21.15.78:0/819612127 conn(0x7fc3fc0f0b70 msgr2=0x7fc3fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:28.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.765+0000 7fc40247e700 1 -- 172.21.15.78:0/819612127 shutdown_connections 2024-01-21T08:10:28.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.766+0000 7fc40247e700 1 -- 172.21.15.78:0/819612127 wait complete. 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.766+0000 7fc40247e700 1 Processor -- start 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.766+0000 7fc40247e700 1 -- start start 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.766+0000 7fc40247e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.766+0000 7fc40247e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc106bd0 con 0x7fc3fc060280 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3fbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3fbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52478/0 (socket says 172.21.15.78:52478) 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3fbfff700 1 -- 172.21.15.78:0/3213373398 learned_addr learned my addr 172.21.15.78:0/3213373398 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3fbfff700 1 -- 172.21.15.78:0/3213373398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec005040 con 0x7fc3fc060280 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3fbfff700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc1066e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc3ec013510 tx=0x7fc3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3f8ff9700 1 -- 172.21.15.78:0/3213373398 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3ec009320 con 0x7fc3fc060280 2024-01-21T08:10:28.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc106dd0 con 0x7fc3fc060280 2024-01-21T08:10:28.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3f8ff9700 1 -- 172.21.15.78:0/3213373398 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3ec013f00 con 0x7fc3fc060280 2024-01-21T08:10:28.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc3f8ff9700 1 -- 172.21.15.78:0/3213373398 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3ec014dc0 con 0x7fc3fc060280 2024-01-21T08:10:28.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.767+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc107260 con 0x7fc3fc060280 2024-01-21T08:10:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.768+0000 7fc3f8ff9700 1 -- 172.21.15.78:0/3213373398 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3ec014420 con 0x7fc3fc060280 2024-01-21T08:10:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.768+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc045ad0 con 0x7fc3fc060280 2024-01-21T08:10:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.768+0000 7fc3f8ff9700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3e402ff60 0x7fc3e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.768+0000 7fc3f8ff9700 1 -- 172.21.15.78:0/3213373398 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3ec041050 con 0x7fc3fc060280 2024-01-21T08:10:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.768+0000 7fc3fb7fe700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3e402ff60 0x7fc3e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.769+0000 7fc3fb7fe700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3e402ff60 0x7fc3e4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc3f00096a0 tx=0x7fc3f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:28.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.773+0000 7fc3f8ff9700 1 -- 172.21.15.78:0/3213373398 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3ec0095f0 con 0x7fc3fc060280 2024-01-21T08:10:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.981+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3fc108110 con 0x7fc3fc060280 2024-01-21T08:10:28.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.981+0000 7fc3f8ff9700 1 -- 172.21.15.78:0/3213373398 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc3ec01cb10 con 0x7fc3fc060280 2024-01-21T08:10:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:28.984 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:28.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3e402ff60 msgr2=0x7fc3e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:28.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3e402ff60 0x7fc3e4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc3f00096a0 tx=0x7fc3f0006b40).stop 2024-01-21T08:10:28.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 msgr2=0x7fc3fc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:28.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc1066e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc3ec013510 tx=0x7fc3ec00b040).stop 2024-01-21T08:10:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 shutdown_connections 2024-01-21T08:10:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc3e402ff60 0x7fc3e4032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 --2- 172.21.15.78:0/3213373398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3fc060280 0x7fc3fc1066e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 >> 172.21.15.78:0/3213373398 conn(0x7fc3fc0f0b70 msgr2=0x7fc3fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 shutdown_connections 2024-01-21T08:10:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:28.983+0000 7fc40247e700 1 -- 172.21.15.78:0/3213373398 wait complete. 2024-01-21T08:10:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:29.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:28.639404+0000 mgr.smithi046.lmxnwc (mgr.14164) 161 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:29.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:28.982487+0000 mon.smithi046 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.78:0/3213373398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:30.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:30.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:10:29.712075+0000 mgr.smithi046.lmxnwc (mgr.14164) 162 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi046 ... 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:08:37Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:09:14Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:09:52Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:10:30.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('uid/gid not found') 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: uid/gid not found 2024-01-21T08:10:30.992 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: yield (conn, connr) 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: code, '\n'.join(err))) 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi046 ... 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi046 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:08:37Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:09:14Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:09:52Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:10:30.993 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('uid/gid not found') 2024-01-21T08:10:30.994 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: uid/gid not found 2024-01-21T08:10:30.995 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:29.712738+0000 mon.smithi046 (mon.0) 237 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:10:30.995 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:29.934130+0000 mon.smithi046 (mon.0) 238 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:30.995 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:30.166941+0000 mon.smithi046 (mon.0) 239 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.959+0000 7ff343fff700 1 -- 172.21.15.78:0/159277362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 msgr2=0x7ff3440f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.959+0000 7ff343fff700 1 --2- 172.21.15.78:0/159277362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3440f7d00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7ff32c009fb0 tx=0x7ff32c009b30).stop 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.959+0000 7ff343fff700 1 -- 172.21.15.78:0/159277362 shutdown_connections 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.959+0000 7ff343fff700 1 --2- 172.21.15.78:0/159277362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3440f7d00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.959+0000 7ff343fff700 1 -- 172.21.15.78:0/159277362 >> 172.21.15.78:0/159277362 conn(0x7ff3440f0120 msgr2=0x7ff3440f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.960+0000 7ff343fff700 1 -- 172.21.15.78:0/159277362 shutdown_connections 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.960+0000 7ff343fff700 1 -- 172.21.15.78:0/159277362 wait complete. 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.960+0000 7ff343fff700 1 Processor -- start 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.960+0000 7ff343fff700 1 -- start start 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff343fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3441064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff343fff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3441069d0 con 0x7ff3440f78e0 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff342ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3441064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff342ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3441064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:52490/0 (socket says 172.21.15.78:52490) 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff342ffd700 1 -- 172.21.15.78:0/307407329 learned_addr learned my addr 172.21.15.78:0/307407329 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff342ffd700 1 -- 172.21.15.78:0/307407329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c005040 con 0x7ff3440f78e0 2024-01-21T08:10:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff342ffd700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3441064e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff32c009830 tx=0x7ff32c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff33b7fe700 1 -- 172.21.15.78:0/307407329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff32c009320 con 0x7ff3440f78e0 2024-01-21T08:10:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff344106bd0 con 0x7ff3440f78e0 2024-01-21T08:10:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff33b7fe700 1 -- 172.21.15.78:0/307407329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff32c0136c0 con 0x7ff3440f78e0 2024-01-21T08:10:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff33b7fe700 1 -- 172.21.15.78:0/307407329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff32c01ee90 con 0x7ff3440f78e0 2024-01-21T08:10:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.961+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff344107060 con 0x7ff3440f78e0 2024-01-21T08:10:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.962+0000 7ff33b7fe700 1 -- 172.21.15.78:0/307407329 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff32c01e420 con 0x7ff3440f78e0 2024-01-21T08:10:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.962+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff344045ad0 con 0x7ff3440f78e0 2024-01-21T08:10:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.962+0000 7ff33b7fe700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff33002ff10 0x7ff3300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.963+0000 7ff33b7fe700 1 -- 172.21.15.78:0/307407329 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff32c040f70 con 0x7ff3440f78e0 2024-01-21T08:10:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.963+0000 7ff3427fc700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff33002ff10 0x7ff3300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.963+0000 7ff3427fc700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff33002ff10 0x7ff3300323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff3340096a0 tx=0x7ff334006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:31.967+0000 7ff33b7fe700 1 -- 172.21.15.78:0/307407329 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff32c06d2b0 con 0x7ff3440f78e0 2024-01-21T08:10:32.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.192+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff344107f10 con 0x7ff3440f78e0 2024-01-21T08:10:32.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.192+0000 7ff33b7fe700 1 -- 172.21.15.78:0/307407329 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff32c02c2d0 con 0x7ff3440f78e0 2024-01-21T08:10:32.195 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:32.195 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:32.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff33002ff10 msgr2=0x7ff3300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:32.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff33002ff10 0x7ff3300323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff3340096a0 tx=0x7ff334006b40).stop 2024-01-21T08:10:32.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 msgr2=0x7ff3441064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:32.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3441064e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff32c009830 tx=0x7ff32c00b040).stop 2024-01-21T08:10:32.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 shutdown_connections 2024-01-21T08:10:32.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff33002ff10 0x7ff3300323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:32.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 --2- 172.21.15.78:0/307407329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3440f78e0 0x7ff3441064e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:32.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 >> 172.21.15.78:0/307407329 conn(0x7ff3440f0120 msgr2=0x7ff3440f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:32.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 shutdown_connections 2024-01-21T08:10:32.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:32.194+0000 7ff343fff700 1 -- 172.21.15.78:0/307407329 wait complete. 2024-01-21T08:10:32.199 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:30.639750+0000 mgr.smithi046.lmxnwc (mgr.14164) 163 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:31.259965+0000 mon.smithi046 (mon.0) 240 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:32.194249+0000 mon.smithi046 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.78:0/307407329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:33.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:33.823 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:32.640133+0000 mgr.smithi046.lmxnwc (mgr.14164) 164 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:33.049249+0000 mon.smithi046 (mon.0) 242 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:33.318332+0000 mon.smithi046 (mon.0) 243 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:33.319051+0000 mgr.smithi046.lmxnwc (mgr.14164) 165 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:33.321202+0000 mon.smithi046 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:33.322592+0000 mon.smithi046 (mon.0) 245 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:33.324567+0000 mon.smithi046 (mon.0) 246 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:33.325405+0000 mon.smithi046 (mon.0) 247 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:10:34.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:10:33.326174+0000 mgr.smithi046.lmxnwc (mgr.14164) 166 : cephadm [INF] Deploying daemon crash.smithi078 on smithi078 2024-01-21T08:10:35.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.553+0000 7f3b72694700 1 -- 172.21.15.78:0/1416985389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 msgr2=0x7f3b6c0fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.553+0000 7f3b72694700 1 --2- 172.21.15.78:0/1416985389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c0fb3a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f3b68009fb0 tx=0x7f3b68009b30).stop 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.553+0000 7f3b72694700 1 -- 172.21.15.78:0/1416985389 shutdown_connections 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.553+0000 7f3b72694700 1 --2- 172.21.15.78:0/1416985389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c0fb3a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.553+0000 7f3b72694700 1 -- 172.21.15.78:0/1416985389 >> 172.21.15.78:0/1416985389 conn(0x7f3b6c0f09a0 msgr2=0x7f3b6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.553+0000 7f3b72694700 1 -- 172.21.15.78:0/1416985389 shutdown_connections 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.553+0000 7f3b72694700 1 -- 172.21.15.78:0/1416985389 wait complete. 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.554+0000 7f3b72694700 1 Processor -- start 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.554+0000 7f3b72694700 1 -- start start 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.554+0000 7f3b72694700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c1086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.554+0000 7f3b72694700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c108be0 con 0x7f3b6c060160 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.554+0000 7f3b71692700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c1086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.554+0000 7f3b71692700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c1086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:46012/0 (socket says 172.21.15.78:46012) 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.554+0000 7f3b71692700 1 -- 172.21.15.78:0/3913842595 learned_addr learned my addr 172.21.15.78:0/3913842595 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.555+0000 7f3b71692700 1 -- 172.21.15.78:0/3913842595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b68005040 con 0x7f3b6c060160 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.555+0000 7f3b71692700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c1086f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3b68013510 tx=0x7f3b6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.555+0000 7f3b61ffb700 1 -- 172.21.15.78:0/3913842595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b68009320 con 0x7f3b6c060160 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.555+0000 7f3b61ffb700 1 -- 172.21.15.78:0/3913842595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b68013f00 con 0x7f3b6c060160 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.555+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c108de0 con 0x7f3b6c060160 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.555+0000 7f3b61ffb700 1 -- 172.21.15.78:0/3913842595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b68014d90 con 0x7f3b6c060160 2024-01-21T08:10:35.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.555+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c109270 con 0x7f3b6c060160 2024-01-21T08:10:35.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.556+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c045ad0 con 0x7f3b6c060160 2024-01-21T08:10:35.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.556+0000 7f3b61ffb700 1 -- 172.21.15.78:0/3913842595 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b68014420 con 0x7f3b6c060160 2024-01-21T08:10:35.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.556+0000 7f3b61ffb700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3b5802ffb0 0x7f3b58032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.556+0000 7f3b61ffb700 1 -- 172.21.15.78:0/3913842595 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b68040ff0 con 0x7f3b6c060160 2024-01-21T08:10:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.557+0000 7f3b70e91700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3b5802ffb0 0x7f3b58032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.557+0000 7f3b70e91700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3b5802ffb0 0x7f3b58032470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3b5c0096a0 tx=0x7f3b5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:35.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.561+0000 7f3b61ffb700 1 -- 172.21.15.78:0/3913842595 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b6806d2b0 con 0x7f3b6c060160 2024-01-21T08:10:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:34.317095+0000 mon.smithi046 (mon.0) 248 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-01-21T08:10:35.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.778+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b6c10a0c0 con 0x7f3b6c060160 2024-01-21T08:10:35.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.779+0000 7f3b61ffb700 1 -- 172.21.15.78:0/3913842595 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b680095f0 con 0x7f3b6c060160 2024-01-21T08:10:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:35.782 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3b5802ffb0 msgr2=0x7f3b58032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3b5802ffb0 0x7f3b58032470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3b5c0096a0 tx=0x7f3b5c006b40).stop 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 msgr2=0x7f3b6c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c1086f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3b68013510 tx=0x7f3b6800b040).stop 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 shutdown_connections 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3b5802ffb0 0x7f3b58032470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 --2- 172.21.15.78:0/3913842595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b6c060160 0x7f3b6c1086f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 >> 172.21.15.78:0/3913842595 conn(0x7f3b6c0f09a0 msgr2=0x7f3b6c0f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 shutdown_connections 2024-01-21T08:10:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:35.782+0000 7f3b72694700 1 -- 172.21.15.78:0/3913842595 wait complete. 2024-01-21T08:10:35.786 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:36.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:35.319432+0000 mgr.smithi046.lmxnwc (mgr.14164) 167 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:36.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:35.622927+0000 mon.smithi046 (mon.0) 249 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:36.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:35.624427+0000 mon.smithi046 (mon.0) 250 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi078.kcfjgo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-21T08:10:36.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:35.626424+0000 mon.smithi046 (mon.0) 251 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi078.kcfjgo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-21T08:10:36.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:35.627302+0000 mon.smithi046 (mon.0) 252 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-21T08:10:36.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:35.628284+0000 mon.smithi046 (mon.0) 253 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:10:36.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:35.781089+0000 mon.smithi046 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.78:0/3913842595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:37.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:37.178 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:10:35.629227+0000 mgr.smithi046.lmxnwc (mgr.14164) 168 : cephadm [INF] Deploying daemon mgr.smithi078.kcfjgo on smithi078 2024-01-21T08:10:38.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 -- 172.21.15.78:0/1807429985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 msgr2=0x7f087c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:38.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 --2- 172.21.15.78:0/1807429985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c0f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f086c009fb0 tx=0x7f086c009b30).stop 2024-01-21T08:10:38.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 -- 172.21.15.78:0/1807429985 shutdown_connections 2024-01-21T08:10:38.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 --2- 172.21.15.78:0/1807429985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c0f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:38.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 -- 172.21.15.78:0/1807429985 >> 172.21.15.78:0/1807429985 conn(0x7f087c0f1c70 msgr2=0x7f087c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 -- 172.21.15.78:0/1807429985 shutdown_connections 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 -- 172.21.15.78:0/1807429985 wait complete. 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 Processor -- start 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.618+0000 7f088173b700 1 -- start start 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f088173b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f088173b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c106c20 con 0x7f087c0f6510 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f087affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f087affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.78:46032/0 (socket says 172.21.15.78:46032) 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f087affd700 1 -- 172.21.15.78:0/3446358761 learned_addr learned my addr 172.21.15.78:0/3446358761 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f087affd700 1 -- 172.21.15.78:0/3446358761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c005040 con 0x7f087c0f6510 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f087affd700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f086c013510 tx=0x7f086c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f0873fff700 1 -- 172.21.15.78:0/3446358761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c009320 con 0x7f087c0f6510 2024-01-21T08:10:38.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c106e20 con 0x7f087c0f6510 2024-01-21T08:10:38.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f0873fff700 1 -- 172.21.15.78:0/3446358761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f086c013f00 con 0x7f087c0f6510 2024-01-21T08:10:38.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f0873fff700 1 -- 172.21.15.78:0/3446358761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c014d90 con 0x7f087c0f6510 2024-01-21T08:10:38.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.619+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c1072b0 con 0x7f087c0f6510 2024-01-21T08:10:38.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.620+0000 7f0873fff700 1 -- 172.21.15.78:0/3446358761 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f086c0263a0 con 0x7f087c0f6510 2024-01-21T08:10:38.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.621+0000 7f0873fff700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f086802ff60 0x7f0868032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:38.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.620+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c045ad0 con 0x7f087c0f6510 2024-01-21T08:10:38.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.621+0000 7f0873fff700 1 -- 172.21.15.78:0/3446358761 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f086c0412f0 con 0x7f087c0f6510 2024-01-21T08:10:38.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.621+0000 7f08727fc700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f086802ff60 0x7f0868032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:38.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.621+0000 7f08727fc700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f086802ff60 0x7f0868032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f08640096a0 tx=0x7f0864006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.625+0000 7f0873fff700 1 -- 172.21.15.78:0/3446358761 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f086c012510 con 0x7f087c0f6510 2024-01-21T08:10:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:37.319735+0000 mgr.smithi046.lmxnwc (mgr.14164) 169 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:38.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:37.320170+0000 mon.smithi046 (mon.0) 255 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:10:38.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:37.321341+0000 mon.smithi046 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-21T08:10:38.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:37.322205+0000 mon.smithi046 (mon.0) 257 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:10:38.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:10:37.322904+0000 mgr.smithi046.lmxnwc (mgr.14164) 170 : cephadm [INF] Deploying daemon mon.smithi078 on smithi078 2024-01-21T08:10:38.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.854+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f087c0fc050 con 0x7f087c0f6510 2024-01-21T08:10:38.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.855+0000 7f0873fff700 1 -- 172.21.15.78:0/3446358761 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f086c012330 con 0x7f087c0f6510 2024-01-21T08:10:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:38.857 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:04:24.344932Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f086802ff60 msgr2=0x7f0868032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f086802ff60 0x7f0868032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f08640096a0 tx=0x7f0864006b40).stop 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 msgr2=0x7f087c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f086c013510 tx=0x7f086c00b040).stop 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 shutdown_connections 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f086802ff60 0x7f0868032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 --2- 172.21.15.78:0/3446358761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f087c0f6510 0x7f087c106730 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 >> 172.21.15.78:0/3446358761 conn(0x7f087c0f1c70 msgr2=0x7f087c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:38.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 shutdown_connections 2024-01-21T08:10:38.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:38.857+0000 7f088173b700 1 -- 172.21.15.78:0/3446358761 wait complete. 2024-01-21T08:10:38.860 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 1 2024-01-21T08:10:39.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:38.856218+0000 mon.smithi046 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.78:0/3446358761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:39.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:39 smithi078 systemd[1]: Starting Ceph mon.smithi078 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:10:40.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T08:10:40.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T08:10:40.418 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.100+0000 7f36cd598700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T08:10:40.418 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.100+0000 7f36cd598700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T08:10:40.418 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.100+0000 7f36cd598700 0 pidfile_write: ignore empty --pid-file 2024-01-21T08:10:40.419 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.104+0000 7f36cd598700 0 load: jerasure load: lrc load: isa 2024-01-21T08:10:40.419 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T08:10:40.419 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.419 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T08:10:40.419 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Compile date May 13 2021 2024-01-21T08:10:40.419 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: DB SUMMARY 2024-01-21T08:10:40.420 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.420 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T08:10:40.420 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.420 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T08:10:40.420 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.420 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi078/store.db dir, Total Num: 0, files: 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi078/store.db: 000003.log size: 511 ; 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.env: 0x55d50e2451c0 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.fs: Posix File System 2024-01-21T08:10:40.421 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.info_log: 0x55d5100d5960 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.statistics: (nil) 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.use_fsync: 0 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T08:10:40.422 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.db_log_dir: 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi078/store.db 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T08:10:40.423 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.write_buffer_manager: 0x55d510d4a030 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T08:10:40.424 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.unordered_write: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.row_cache: None 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.wal_filter: None 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T08:10:40.425 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T08:10:40.426 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T08:10:40.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T08:10:40.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T08:10:40.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T08:10:40.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T08:10:40.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T08:10:40.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_open_files: -1 2024-01-21T08:10:40.428 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Compression algorithms supported: 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kZSTD supported: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T08:10:40.429 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi078/store.db/MANIFEST-000001 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.merge_operator: 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_filter: None 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T08:10:40.430 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T08:10:40.431 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d50ffec128) 2024-01-21T08:10:40.431 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cache_index_and_filter_blocks: 1 2024-01-21T08:10:40.431 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T08:10:40.431 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T08:10:40.431 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: pin_top_level_index_and_filter: 1 2024-01-21T08:10:40.432 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: index_type: 0 2024-01-21T08:10:40.432 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: data_block_index_type: 0 2024-01-21T08:10:40.432 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: index_shortening: 1 2024-01-21T08:10:40.432 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T08:10:40.432 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: hash_index_allow_collision: 1 2024-01-21T08:10:40.432 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: checksum: 1 2024-01-21T08:10:40.433 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: no_block_cache: 0 2024-01-21T08:10:40.433 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_cache: 0x55d510022f10 2024-01-21T08:10:40.433 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_cache_name: BinnedLRUCache 2024-01-21T08:10:40.433 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_cache_options: 2024-01-21T08:10:40.433 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: capacity : 536870912 2024-01-21T08:10:40.434 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: num_shard_bits : 4 2024-01-21T08:10:40.434 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: strict_capacity_limit : 0 2024-01-21T08:10:40.434 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: high_pri_pool_ratio: 0.000 2024-01-21T08:10:40.434 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_cache_compressed: (nil) 2024-01-21T08:10:40.434 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: persistent_cache: (nil) 2024-01-21T08:10:40.435 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_size: 4096 2024-01-21T08:10:40.435 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_size_deviation: 10 2024-01-21T08:10:40.435 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_restart_interval: 16 2024-01-21T08:10:40.435 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: index_block_restart_interval: 1 2024-01-21T08:10:40.435 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: metadata_block_size: 4096 2024-01-21T08:10:40.435 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: partition_filters: 0 2024-01-21T08:10:40.435 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: use_delta_encoding: 1 2024-01-21T08:10:40.436 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T08:10:40.436 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: whole_key_filtering: 1 2024-01-21T08:10:40.436 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: verify_compression: 0 2024-01-21T08:10:40.436 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: read_amp_bytes_per_bit: 0 2024-01-21T08:10:40.436 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: format_version: 2 2024-01-21T08:10:40.437 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: enable_index_compression: 1 2024-01-21T08:10:40.437 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: block_align: 0 2024-01-21T08:10:40.437 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.437 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T08:10:40.437 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T08:10:40.438 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compression: NoCompression 2024-01-21T08:10:40.438 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T08:10:40.438 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T08:10:40.438 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T08:10:40.438 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.num_levels: 7 2024-01-21T08:10:40.438 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T08:10:40.438 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T08:10:40.439 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T08:10:40.440 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T08:10:40.441 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T08:10:40.442 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.ttl: 2592000 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.105+0000 7f36cd598700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.106+0000 7f36cd598700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi078/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.106+0000 7f36cd598700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.443 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.106+0000 7f36cd598700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705824640107180, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.106+0000 7f36cd598700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.106+0000 7f36cd598700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 bash[57621]: 505d28f67929fcd1d2751a5546ca336b5e04c33d068f781b2a6bcf12aeb1759a 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.106+0000 7f36cd598700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705824640107656, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705824640, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.106+0000 7f36cd598700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.107+0000 7f36cd598700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705824640108034, "job": 1, "event": "recovery_finished"} 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.107+0000 7f36cd598700 4 rocksdb: DB pointer 0x55d5100e7800 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.107+0000 7f36b43b0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.107+0000 7f36b43b0700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ** DB Stats ** 2024-01-21T08:10:40.444 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ** Compaction Stats [default] ** 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-01-21T08:10:40.445 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ** Compaction Stats [default] ** 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:10:40.446 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ** Compaction Stats [default] ** 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ** Compaction Stats [default] ** 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T08:10:40.447 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.448 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.108+0000 7f36cd598700 0 mon.smithi078 does not exist in monmap, will attempt to join an existing cluster 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.108+0000 7f36cd598700 0 using public_addr v2:172.21.15.78:0/0 -> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.108+0000 7f36cd598700 0 starting mon.smithi078 rank -1 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi078 fsid a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.109+0000 7f36cd598700 1 mon.smithi078@-1(???) e0 preinit fsid a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 systemd[1]: Started Ceph mon.smithi078 for a17ce476-b833-11ee-95b0-87774f69a715. 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.166+0000 7f36b6bb5700 0 mon.smithi078@-1(synchronizing).mds e1 new map 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.166+0000 7f36b6bb5700 0 mon.smithi078@-1(synchronizing).mds e1 print_map 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: e1 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T08:10:40.449 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: legacy client fscid: -1 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: No filesystems configured 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.166+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.166+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.166+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 0 mon.smithi078@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 0 mon.smithi078@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 0 mon.smithi078@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T08:10:40.450 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 0 mon.smithi078@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:38.856218+0000 mon.smithi046 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.78:0/3446358761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.167+0000 7f36b6bb5700 1 mon.smithi078@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expand map: {default=false} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta from 'false' to 'false' 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expanded map: {default=false} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expand map: {default=info} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta from 'info' to 'info' 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expanded map: {default=info} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expand map: {default=daemon} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expand map: {default=debug} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 20 expand_channel_meta expanded map: {default=debug} 2024-01-21T08:10:40.451 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:10:40.168+0000 7f36b6bb5700 10 mon.smithi078@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-01-21T08:10:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:10:40.172301+0000 mgr.smithi046.lmxnwc (mgr.14164) 172 : cephadm [INF] Deploying daemon node-exporter.smithi078 on smithi078 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:40.174203+0000 mon.smithi046 (mon.0) 261 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:40.174410+0000 mon.smithi046 (mon.0) 262 : cluster [INF] mon.smithi046 calling monitor election 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:40.187687+0000 mon.smithi046 (mon.0) 263 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:41.171856+0000 mon.smithi046 (mon.0) 264 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:41.320312+0000 mgr.smithi046.lmxnwc (mgr.14164) 173 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:42.171900+0000 mon.smithi046 (mon.0) 265 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:42.172984+0000 mon.smithi078 (mon.1) 1 : cluster [INF] mon.smithi078 calling monitor election 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:42.455536+0000 mon.smithi046 (mon.0) 266 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi078.kcfjgo/crt"}]: dispatch 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:43.171956+0000 mon.smithi046 (mon.0) 267 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:43.320717+0000 mgr.smithi046.lmxnwc (mgr.14164) 174 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:44.172172+0000 mon.smithi046 (mon.0) 268 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:45.172182+0000 mon.smithi046 (mon.0) 269 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.200995+0000 mon.smithi046 (mon.0) 270 : cluster [INF] mon.smithi046 is new leader, mons smithi046,smithi078 in quorum (ranks 0,1) 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.203983+0000 mon.smithi046 (mon.0) 271 : cluster [DBG] monmap e2: 2 mons at {smithi046=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],smithi078=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.204042+0000 mon.smithi046 (mon.0) 272 : cluster [DBG] fsmap 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.204078+0000 mon.smithi046 (mon.0) 273 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.204298+0000 mon.smithi046 (mon.0) 274 : cluster [DBG] mgrmap e17: smithi046.lmxnwc(active, since 5m) 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.205232+0000 mon.smithi046 (mon.0) 275 : cluster [DBG] Standby manager daemon smithi078.kcfjgo started 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:45.206284+0000 mon.smithi046 (mon.0) 276 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.207890+0000 mon.smithi046 (mon.0) 277 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.207919+0000 mon.smithi046 (mon.0) 278 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.207930+0000 mon.smithi046 (mon.0) 279 : cluster [WRN] daemon node-exporter.smithi046 on smithi046 is in unknown state 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:45.208369+0000 mon.smithi046 (mon.0) 280 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi078.kcfjgo/key"}]: dispatch 2024-01-21T08:10:45.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:45.208871+0000 mon.smithi046 (mon.0) 281 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-21T08:10:45.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:10:40.172301+0000 mgr.smithi046.lmxnwc (mgr.14164) 172 : cephadm [INF] Deploying daemon node-exporter.smithi078 on smithi078 2024-01-21T08:10:45.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:40.174203+0000 mon.smithi046 (mon.0) 261 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi046"}]: dispatch 2024-01-21T08:10:45.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:40.174410+0000 mon.smithi046 (mon.0) 262 : cluster [INF] mon.smithi046 calling monitor election 2024-01-21T08:10:45.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:40.187687+0000 mon.smithi046 (mon.0) 263 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:41.171856+0000 mon.smithi046 (mon.0) 264 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:41.320312+0000 mgr.smithi046.lmxnwc (mgr.14164) 173 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:42.171900+0000 mon.smithi046 (mon.0) 265 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:42.172984+0000 mon.smithi078 (mon.1) 1 : cluster [INF] mon.smithi078 calling monitor election 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:42.455536+0000 mon.smithi046 (mon.0) 266 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi078.kcfjgo/crt"}]: dispatch 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:43.171956+0000 mon.smithi046 (mon.0) 267 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:43.320717+0000 mgr.smithi046.lmxnwc (mgr.14164) 174 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:44.172172+0000 mon.smithi046 (mon.0) 268 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:45.172182+0000 mon.smithi046 (mon.0) 269 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.200995+0000 mon.smithi046 (mon.0) 270 : cluster [INF] mon.smithi046 is new leader, mons smithi046,smithi078 in quorum (ranks 0,1) 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.203983+0000 mon.smithi046 (mon.0) 271 : cluster [DBG] monmap e2: 2 mons at {smithi046=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],smithi078=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.204042+0000 mon.smithi046 (mon.0) 272 : cluster [DBG] fsmap 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.204078+0000 mon.smithi046 (mon.0) 273 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.204298+0000 mon.smithi046 (mon.0) 274 : cluster [DBG] mgrmap e17: smithi046.lmxnwc(active, since 5m) 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.205232+0000 mon.smithi046 (mon.0) 275 : cluster [DBG] Standby manager daemon smithi078.kcfjgo started 2024-01-21T08:10:45.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:45.206284+0000 mon.smithi046 (mon.0) 276 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-21T08:10:45.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.207890+0000 mon.smithi046 (mon.0) 277 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-01-21T08:10:45.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.207919+0000 mon.smithi046 (mon.0) 278 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-01-21T08:10:45.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.207930+0000 mon.smithi046 (mon.0) 279 : cluster [WRN] daemon node-exporter.smithi046 on smithi046 is in unknown state 2024-01-21T08:10:45.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:45.208369+0000 mon.smithi046 (mon.0) 280 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi078.kcfjgo/key"}]: dispatch 2024-01-21T08:10:45.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:45.208871+0000 mon.smithi046 (mon.0) 281 : audit [DBG] from='mgr.? 172.21.15.78:0/2665783752' entity='mgr.smithi078.kcfjgo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-21T08:10:45.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.888+0000 7f1175ea5700 1 -- 172.21.15.78:0/4077121049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1158003250 msgr2=0x7f11580056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:45.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.888+0000 7f1175ea5700 1 --2- 172.21.15.78:0/4077121049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1158003250 0x7f11580056e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f11700f61c0 tx=0x7f1160002040).stop 2024-01-21T08:10:45.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.888+0000 7f1175ea5700 1 -- 172.21.15.78:0/4077121049 shutdown_connections 2024-01-21T08:10:45.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.888+0000 7f1175ea5700 1 --2- 172.21.15.78:0/4077121049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1158003250 0x7f11580056e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:45.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.888+0000 7f1175ea5700 1 -- 172.21.15.78:0/4077121049 >> 172.21.15.78:0/4077121049 conn(0x7f11700f0b70 msgr2=0x7f11700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:45.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.888+0000 7f1175ea5700 1 -- 172.21.15.78:0/4077121049 shutdown_connections 2024-01-21T08:10:45.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.888+0000 7f1175ea5700 1 -- 172.21.15.78:0/4077121049 wait complete. 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.889+0000 7f1175ea5700 1 Processor -- start 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.889+0000 7f1175ea5700 1 -- start start 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.889+0000 7f1175ea5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 0x7f1170106680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.889+0000 7f1175ea5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1170106b70 0x7f1170107a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f1175ea5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11701070d0 con 0x7f1170106b70 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f1175ea5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170107210 con 0x7f1158003250 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 0x7f1170106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 0x7f1170106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35076/0 (socket says 172.21.15.78:35076) 2024-01-21T08:10:45.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116f7fe700 1 -- 172.21.15.78:0/1220972446 learned_addr learned my addr 172.21.15.78:0/1220972446 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116f7fe700 1 -- 172.21.15.78:0/1220972446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 msgr2=0x7f1170106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116f7fe700 1 -- 172.21.15.78:0/1220972446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 msgr2=0x7f1170106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116f7fe700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 0x7f1170106680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116f7fe700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 0x7f1170106680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116effd700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1170106b70 0x7f1170107a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116effd700 1 -- 172.21.15.78:0/1220972446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 msgr2=0x7f1170106680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116effd700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 0x7f1170106680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116effd700 1 -- 172.21.15.78:0/1220972446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160005040 con 0x7f1170106b70 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.890+0000 7f116effd700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1170106b70 0x7f1170107a60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f1164009390 tx=0x7f1164006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.891+0000 7f1157fff700 1 -- 172.21.15.78:0/1220972446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11640115e0 con 0x7f1170106b70 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.891+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1170108070 con 0x7f1170106b70 2024-01-21T08:10:45.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.891+0000 7f1157fff700 1 -- 172.21.15.78:0/1220972446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1164011bf0 con 0x7f1170106b70 2024-01-21T08:10:45.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.891+0000 7f1157fff700 1 -- 172.21.15.78:0/1220972446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1164010bf0 con 0x7f1170106b70 2024-01-21T08:10:45.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.891+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1170108630 con 0x7f1170106b70 2024-01-21T08:10:45.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.892+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1170059100 con 0x7f1170106b70 2024-01-21T08:10:45.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.892+0000 7f1157fff700 1 -- 172.21.15.78:0/1220972446 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1164018030 con 0x7f1170106b70 2024-01-21T08:10:45.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.893+0000 7f1157fff700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f115c05c3a0 0x7f115c05e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:45.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.893+0000 7f1157fff700 1 -- 172.21.15.78:0/1220972446 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1164074b80 con 0x7f1170106b70 2024-01-21T08:10:45.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.893+0000 7f116f7fe700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f115c05c3a0 0x7f115c05e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:45.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.893+0000 7f116f7fe700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f115c05c3a0 0x7f115c05e860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1160004560 tx=0x7f1160002040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:45.897+0000 7f1157fff700 1 -- 172.21.15.78:0/1220972446 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11640452b0 con 0x7f1170106b70 2024-01-21T08:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.106+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1170045ad0 con 0x7f1170106b70 2024-01-21T08:10:46.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.107+0000 7f1157fff700 1 -- 172.21.15.78:0/1220972446 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+964 (secure 0 0 0) 0x7f1164016020 con 0x7f1170106b70 2024-01-21T08:10:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-01-21T08:10:46.109 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","modified":"2024-01-21T08:10:40.171934Z","created":"2024-01-21T08:04:24.344932Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi046","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi078","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f115c05c3a0 msgr2=0x7f115c05e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f115c05c3a0 0x7f115c05e860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1160004560 tx=0x7f1160002040).stop 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1170106b70 msgr2=0x7f1170107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1170106b70 0x7f1170107a60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f1164009390 tx=0x7f1164006e20).stop 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 shutdown_connections 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f115c05c3a0 0x7f115c05e860 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1158003250 0x7f1170106680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 --2- 172.21.15.78:0/1220972446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1170106b70 0x7f1170107a60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:46.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 >> 172.21.15.78:0/1220972446 conn(0x7f11700f0b70 msgr2=0x7f117005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:46.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 shutdown_connections 2024-01-21T08:10:46.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:10:46.109+0000 7f1175ea5700 1 -- 172.21.15.78:0/1220972446 wait complete. 2024-01-21T08:10:46.112 INFO:teuthology.orchestra.run.smithi078.stderr:dumped monmap epoch 2 2024-01-21T08:10:46.456 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:46 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.264714+0000 mon.smithi046 (mon.0) 282 : cluster [DBG] mgrmap e18: smithi046.lmxnwc(active, since 5m), standbys: smithi078.kcfjgo 2024-01-21T08:10:46.456 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:46 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:45.264862+0000 mon.smithi046 (mon.0) 283 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr metadata", "who": "smithi078.kcfjgo", "id": "smithi078.kcfjgo"}]: dispatch 2024-01-21T08:10:46.457 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:46 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:45.321073+0000 mgr.smithi046.lmxnwc (mgr.14164) 175 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:46.457 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:46 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:46.107965+0000 mon.smithi046 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.78:0/1220972446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:46.457 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:46 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:46.172346+0000 mon.smithi046 (mon.0) 285 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:46.490 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-21T08:10:46.490 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph config generate-minimal-conf 2024-01-21T08:10:46.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.264714+0000 mon.smithi046 (mon.0) 282 : cluster [DBG] mgrmap e18: smithi046.lmxnwc(active, since 5m), standbys: smithi078.kcfjgo 2024-01-21T08:10:46.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:45.264862+0000 mon.smithi046 (mon.0) 283 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr metadata", "who": "smithi078.kcfjgo", "id": "smithi078.kcfjgo"}]: dispatch 2024-01-21T08:10:46.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:45.321073+0000 mgr.smithi046.lmxnwc (mgr.14164) 175 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:46.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:46.107965+0000 mon.smithi046 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.78:0/1220972446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T08:10:46.525 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:46.172346+0000 mon.smithi046 (mon.0) 285 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mon metadata", "id": "smithi078"}]: dispatch 2024-01-21T08:10:47.658 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:47.321425+0000 mgr.smithi046.lmxnwc (mgr.14164) 176 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:47.321425+0000 mgr.smithi046.lmxnwc (mgr.14164) 176 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:47.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.845+0000 7fd151199700 1 -- 172.21.15.46:0/2168397413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 msgr2=0x7fd14c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:47.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.845+0000 7fd151199700 1 --2- 172.21.15.46:0/2168397413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c0f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fd13c009fb0 tx=0x7fd13c009b30).stop 2024-01-21T08:10:47.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.845+0000 7fd151199700 1 -- 172.21.15.46:0/2168397413 shutdown_connections 2024-01-21T08:10:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.845+0000 7fd151199700 1 --2- 172.21.15.46:0/2168397413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c0f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.845+0000 7fd151199700 1 -- 172.21.15.46:0/2168397413 >> 172.21.15.46:0/2168397413 conn(0x7fd14c0f0b70 msgr2=0x7fd14c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.845+0000 7fd151199700 1 -- 172.21.15.46:0/2168397413 shutdown_connections 2024-01-21T08:10:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.845+0000 7fd151199700 1 -- 172.21.15.46:0/2168397413 wait complete. 2024-01-21T08:10:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.846+0000 7fd151199700 1 Processor -- start 2024-01-21T08:10:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.846+0000 7fd151199700 1 -- start start 2024-01-21T08:10:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd151199700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c1056b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd151199700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd14c105ba0 0x7fd14c106a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd151199700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c106100 con 0x7fd14c0f5410 2024-01-21T08:10:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd151199700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c106240 con 0x7fd14c105ba0 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd14ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c1056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd14ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c1056b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47496/0 (socket says 172.21.15.46:47496) 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd14ad9d700 1 -- 172.21.15.46:0/1256456153 learned_addr learned my addr 172.21.15.46:0/1256456153 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd14ad9d700 1 -- 172.21.15.46:0/1256456153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd14c105ba0 msgr2=0x7fd14c106a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd14ad9d700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd14c105ba0 0x7fd14c106a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd14ad9d700 1 -- 172.21.15.46:0/1256456153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c005040 con 0x7fd14c0f5410 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd14ad9d700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c1056b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fd13c002040 tx=0x7fd13c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd1337fe700 1 -- 172.21.15.46:0/1256456153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd13c019070 con 0x7fd14c0f5410 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd1337fe700 1 -- 172.21.15.46:0/1256456153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd13c007c70 con 0x7fd14c0f5410 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14c107040 con 0x7fd14c0f5410 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd1337fe700 1 -- 172.21.15.46:0/1256456153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd13c01d620 con 0x7fd14c0f5410 2024-01-21T08:10:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.847+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c1075a0 con 0x7fd14c0f5410 2024-01-21T08:10:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.848+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c059100 con 0x7fd14c0f5410 2024-01-21T08:10:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.849+0000 7fd1337fe700 1 -- 172.21.15.46:0/1256456153 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd13c025070 con 0x7fd14c0f5410 2024-01-21T08:10:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.849+0000 7fd1337fe700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd13405c3b0 0x7fd13405e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.849+0000 7fd1337fe700 1 -- 172.21.15.46:0/1256456153 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd13c0762c0 con 0x7fd14c0f5410 2024-01-21T08:10:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.849+0000 7fd14a59c700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd13405c3b0 0x7fd13405e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.849+0000 7fd14a59c700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd13405c3b0 0x7fd13405e870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fd1400095b0 tx=0x7fd140006ae0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:47.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:47.853+0000 7fd1337fe700 1 -- 172.21.15.46:0/1256456153 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd13c049f90 con 0x7fd14c0f5410 2024-01-21T08:10:48.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.011+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd14c045ad0 con 0x7fd14c0f5410 2024-01-21T08:10:48.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.012+0000 7fd1337fe700 1 -- 172.21.15.46:0/1256456153 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+223 (secure 0 0 0) 0x7fd13c04d5b0 con 0x7fd14c0f5410 2024-01-21T08:10:48.014 INFO:teuthology.orchestra.run.smithi046.stdout:# minimal ceph.conf for a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:10:48.015 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-01-21T08:10:48.015 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = a17ce476-b833-11ee-95b0-87774f69a715 2024-01-21T08:10:48.015 INFO:teuthology.orchestra.run.smithi046.stdout: mon_host = [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] 2024-01-21T08:10:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd13405c3b0 msgr2=0x7fd13405e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd13405c3b0 0x7fd13405e870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fd1400095b0 tx=0x7fd140006ae0).stop 2024-01-21T08:10:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 msgr2=0x7fd14c1056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c1056b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fd13c002040 tx=0x7fd13c00b040).stop 2024-01-21T08:10:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 shutdown_connections 2024-01-21T08:10:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd13405c3b0 0x7fd13405e870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c0f5410 0x7fd14c1056b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 --2- 172.21.15.46:0/1256456153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd14c105ba0 0x7fd14c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:10:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 >> 172.21.15.46:0/1256456153 conn(0x7fd14c0f0b70 msgr2=0x7fd14c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 shutdown_connections 2024-01-21T08:10:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:48.013+0000 7fd151199700 1 -- 172.21.15.46:0/1256456153 wait complete. 2024-01-21T08:10:48.379 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-21T08:10:48.379 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:10:48.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2024-01-21T08:10:48.413 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:10:48.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T08:10:48.483 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:10:48.483 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2024-01-21T08:10:48.521 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:10:48.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T08:10:48.593 INFO:tasks.cephadm:Deploying OSDs... 2024-01-21T08:10:48.594 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:10:48.594 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-01-21T08:10:48.612 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-21T08:10:48.612 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 164919 Links: 1 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-21 08:06:02.004120554 +0000 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-21 07:59:51.051378564 +0000 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-21 07:59:51.051378564 +0000 2024-01-21T08:10:48.671 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-21T08:10:48.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-21T08:10:48.736 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:48.013655+0000 mon.smithi046 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.46:0/1256456153' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:10:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-21T08:10:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-21T08:10:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.3703e-05 s, 5.5 MB/s 2024-01-21T08:10:48.739 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-21T08:10:48.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:48 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:48.013655+0000 mon.smithi046 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.46:0/1256456153' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:10:48.799 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-01-21T08:10:48.857 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-21T08:10:48.857 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:48.858 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 157537 Links: 1 2024-01-21T08:10:48.858 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:48.858 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:48.858 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-21 08:06:02.004120554 +0000 2024-01-21T08:10:48.858 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-21 07:59:51.399374711 +0000 2024-01-21T08:10:48.858 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-21 07:59:51.399374711 +0000 2024-01-21T08:10:48.858 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-21T08:10:48.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-21T08:10:48.923 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-21T08:10:48.924 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-21T08:10:48.924 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000102519 s, 5.0 MB/s 2024-01-21T08:10:48.926 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-21T08:10:48.984 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-01-21T08:10:49.043 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-21T08:10:49.043 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:49.043 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 167992 Links: 1 2024-01-21T08:10:49.044 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:49.044 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:49.044 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-21 08:06:02.004120554 +0000 2024-01-21T08:10:49.044 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-21 07:59:51.769370614 +0000 2024-01-21T08:10:49.044 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-21 07:59:51.769370614 +0000 2024-01-21T08:10:49.044 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-21T08:10:49.045 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-21T08:10:49.110 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-21T08:10:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-21T08:10:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000100578 s, 5.1 MB/s 2024-01-21T08:10:49.112 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-21T08:10:49.171 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-01-21T08:10:49.229 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 165029 Links: 1 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-21 08:06:02.004120554 +0000 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-21 07:59:52.101366938 +0000 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-21 07:59:52.101366938 +0000 2024-01-21T08:10:49.230 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-21T08:10:49.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-21T08:10:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-21T08:10:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-21T08:10:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000101379 s, 5.1 MB/s 2024-01-21T08:10:49.298 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-21T08:10:49.358 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:10:49.358 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-01-21T08:10:49.380 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-21T08:10:49.381 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 160703 Links: 1 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-01-21 08:10:40.614822282 +0000 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-01-21 07:59:42.955265697 +0000 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-01-21 07:59:42.955265697 +0000 2024-01-21T08:10:49.443 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-01-21T08:10:49.444 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-21T08:10:49.516 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-01-21T08:10:49.516 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-01-21T08:10:49.516 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000281127 s, 1.8 MB/s 2024-01-21T08:10:49.517 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-21T08:10:49.581 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-01-21T08:10:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 155505 Links: 1 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-01-21 08:10:40.614822282 +0000 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-01-21 07:59:43.284259816 +0000 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-01-21 07:59:43.284259816 +0000 2024-01-21T08:10:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-01-21T08:10:49.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-21T08:10:49.716 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-01-21T08:10:49.716 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-01-21T08:10:49.716 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000254713 s, 2.0 MB/s 2024-01-21T08:10:49.717 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-21T08:10:49.780 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-01-21T08:10:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-21T08:10:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:49.845 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 164182 Links: 1 2024-01-21T08:10:49.845 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:49.845 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:49.845 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-01-21 08:10:40.614822282 +0000 2024-01-21T08:10:49.845 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-01-21 07:59:43.630253631 +0000 2024-01-21T08:10:49.845 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-01-21 07:59:43.630253631 +0000 2024-01-21T08:10:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-01-21T08:10:49.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-21T08:10:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-01-21T08:10:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-01-21T08:10:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000205441 s, 2.5 MB/s 2024-01-21T08:10:49.917 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-21T08:10:49.980 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 164247 Links: 1 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-01-21 08:10:40.614822282 +0000 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-01-21 07:59:43.976247447 +0000 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-01-21 07:59:43.976247447 +0000 2024-01-21T08:10:50.043 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-01-21T08:10:50.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-21T08:10:50.114 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-01-21T08:10:50.114 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-01-21T08:10:50.114 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000228316 s, 2.2 MB/s 2024-01-21T08:10:50.115 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-21T08:10:50.179 INFO:tasks.cephadm:Deploying osd.0 on smithi046 with /dev/vg_nvme/lv_4... 2024-01-21T08:10:50.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-21T08:10:50.488 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:49.321634+0000 mgr.smithi046.lmxnwc (mgr.14164) 177 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:49.321634+0000 mgr.smithi046.lmxnwc (mgr.14164) 177 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-21T08:10:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-21T08:10:51.549 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-01-21T08:10:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-01-21T08:10:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00833397 s, 1.3 GB/s 2024-01-21T08:10:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-01-21T08:10:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:51.321878+0000 mgr.smithi046.lmxnwc (mgr.14164) 178 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:51.321878+0000 mgr.smithi046.lmxnwc (mgr.14164) 178 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:51.928 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2024-01-21T08:10:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 -- 172.21.15.46:0/3703045147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f7980 msgr2=0x7f17f80f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/3703045147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f7980 0x7f17f80f7de0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f17e8004660 tx=0x7f17e8009b30).stop 2024-01-21T08:10:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 -- 172.21.15.46:0/3703045147 shutdown_connections 2024-01-21T08:10:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/3703045147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f7980 0x7f17f80f7de0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/3703045147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f80f66e0 0x7f17f80f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 -- 172.21.15.46:0/3703045147 >> 172.21.15.46:0/3703045147 conn(0x7f17f80f1e40 msgr2=0x7f17f80f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 -- 172.21.15.46:0/3703045147 shutdown_connections 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.064+0000 7f17fc8ec700 1 -- 172.21.15.46:0/3703045147 wait complete. 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17fc8ec700 1 Processor -- start 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17fc8ec700 1 -- start start 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17fc8ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f66e0 0x7f17f81091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17fc8ec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f80f7980 0x7f17f8109690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17fc8ec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8109c40 con 0x7f17f80f66e0 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17fc8ec700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8109d80 con 0x7f17f80f7980 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f66e0 0x7f17f81091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f66e0 0x7f17f81091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54518/0 (socket says 172.21.15.46:54518) 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f659c700 1 -- 172.21.15.46:0/766322255 learned_addr learned my addr 172.21.15.46:0/766322255 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f5d9b700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f80f7980 0x7f17f8109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f659c700 1 -- 172.21.15.46:0/766322255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f80f7980 msgr2=0x7f17f8109690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f659c700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f80f7980 0x7f17f8109690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f659c700 1 -- 172.21.15.46:0/766322255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e8005040 con 0x7f17f80f66e0 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17f659c700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f66e0 0x7f17f81091a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f17ec002700 tx=0x7f17ec009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17e6ffd700 1 -- 172.21.15.46:0/766322255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ec009320 con 0x7f17f80f66e0 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17e6ffd700 1 -- 172.21.15.46:0/766322255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17ec00f040 con 0x7f17f80f66e0 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f810a720 con 0x7f17f80f66e0 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.065+0000 7f17e6ffd700 1 -- 172.21.15.46:0/766322255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ec014900 con 0x7f17f80f66e0 2024-01-21T08:10:53.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.066+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f810ace0 con 0x7f17f80f66e0 2024-01-21T08:10:53.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.066+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8045ad0 con 0x7f17f80f66e0 2024-01-21T08:10:53.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.067+0000 7f17e6ffd700 1 -- 172.21.15.46:0/766322255 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f17ec009480 con 0x7f17f80f66e0 2024-01-21T08:10:53.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.067+0000 7f17e6ffd700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f17e005c2b0 0x7f17e005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:10:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.067+0000 7f17e6ffd700 1 -- 172.21.15.46:0/766322255 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17ec075160 con 0x7f17f80f66e0 2024-01-21T08:10:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.067+0000 7f17f5d9b700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f17e005c2b0 0x7f17e005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:10:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.067+0000 7f17f5d9b700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f17e005c2b0 0x7f17e005e770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f17e8009d60 tx=0x7f17e8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:10:53.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.071+0000 7f17e6ffd700 1 -- 172.21.15.46:0/766322255 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17ec048eb0 con 0x7f17f80f66e0 2024-01-21T08:10:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:10:53.254+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f17f810afb0 con 0x7f17e005c2b0 2024-01-21T08:10:53.574 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:53.257697+0000 mon.smithi046 (mon.0) 287 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:10:53.574 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:53.259077+0000 mon.smithi046 (mon.0) 288 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:10:53.575 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:53.259568+0000 mon.smithi046 (mon.0) 289 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:10:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:53.257697+0000 mon.smithi046 (mon.0) 287 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:10:53.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:53.259077+0000 mon.smithi046 (mon.0) 288 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:10:53.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:53.259568+0000 mon.smithi046 (mon.0) 289 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:10:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:53.256921+0000 mgr.smithi046.lmxnwc (mgr.14164) 179 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:10:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:10:53.258873+0000 mgr.smithi046.lmxnwc (mgr.14164) 180 : cephadm [INF] Applying service osd.None on host smithi046... 2024-01-21T08:10:54.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:53.322101+0000 mgr.smithi046.lmxnwc (mgr.14164) 181 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:54.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:53.256921+0000 mgr.smithi046.lmxnwc (mgr.14164) 179 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:10:54.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:10:53.258873+0000 mgr.smithi046.lmxnwc (mgr.14164) 180 : cephadm [INF] Applying service osd.None on host smithi046... 2024-01-21T08:10:54.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:53.322101+0000 mgr.smithi046.lmxnwc (mgr.14164) 181 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:54.850917+0000 mon.smithi046 (mon.0) 290 : audit [INF] from='client.? 172.21.15.46:0/555657113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b512ed9c-653b-46a7-9b39-560a59ab1015"}]: dispatch 2024-01-21T08:10:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:54.854264+0000 mon.smithi046 (mon.0) 291 : audit [INF] from='client.? 172.21.15.46:0/555657113' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b512ed9c-653b-46a7-9b39-560a59ab1015"}]': finished 2024-01-21T08:10:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:54.854304+0000 mon.smithi046 (mon.0) 292 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-21T08:10:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:54.854403+0000 mon.smithi046 (mon.0) 293 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:10:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:10:55.293275+0000 mon.smithi046 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.46:0/509610229' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:10:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:54.850917+0000 mon.smithi046 (mon.0) 290 : audit [INF] from='client.? 172.21.15.46:0/555657113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b512ed9c-653b-46a7-9b39-560a59ab1015"}]: dispatch 2024-01-21T08:10:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:54.854264+0000 mon.smithi046 (mon.0) 291 : audit [INF] from='client.? 172.21.15.46:0/555657113' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b512ed9c-653b-46a7-9b39-560a59ab1015"}]': finished 2024-01-21T08:10:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:54.854304+0000 mon.smithi046 (mon.0) 292 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-21T08:10:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:54.854403+0000 mon.smithi046 (mon.0) 293 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:10:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:10:55.293275+0000 mon.smithi046 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.46:0/509610229' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:10:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:55.322339+0000 mgr.smithi046.lmxnwc (mgr.14164) 182 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:56.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:55.322339+0000 mgr.smithi046.lmxnwc (mgr.14164) 182 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:10:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:57.322734+0000 mgr.smithi046.lmxnwc (mgr.14164) 183 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:10:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:10:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:57.322734+0000 mgr.smithi046.lmxnwc (mgr.14164) 183 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:00.703 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:10:59.323124+0000 mgr.smithi046.lmxnwc (mgr.14164) 184 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:10:59.323124+0000 mgr.smithi046.lmxnwc (mgr.14164) 184 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:01.723 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:00.359981+0000 mon.smithi046 (mon.0) 295 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-21T08:11:01.723 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:00.360592+0000 mon.smithi046 (mon.0) 296 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:00.359981+0000 mon.smithi046 (mon.0) 295 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-21T08:11:01.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:00.360592+0000 mon.smithi046 (mon.0) 296 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:02.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:11:00.361106+0000 mgr.smithi046.lmxnwc (mgr.14164) 185 : cephadm [INF] Deploying daemon osd.0 on smithi046 2024-01-21T08:11:02.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:01.323387+0000 mgr.smithi046.lmxnwc (mgr.14164) 186 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:02.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:11:00.361106+0000 mgr.smithi046.lmxnwc (mgr.14164) 185 : cephadm [INF] Deploying daemon osd.0 on smithi046 2024-01-21T08:11:02.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:01.323387+0000 mgr.smithi046.lmxnwc (mgr.14164) 186 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:02.796 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 0 on host 'smithi046' 2024-01-21T08:11:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.792+0000 7f17e6ffd700 1 -- 172.21.15.46:0/766322255 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f17f810afb0 con 0x7f17e005c2b0 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f17e005c2b0 msgr2=0x7f17e005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f17e005c2b0 0x7f17e005e770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f17e8009d60 tx=0x7f17e8009b30).stop 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f66e0 msgr2=0x7f17f81091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f66e0 0x7f17f81091a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f17ec002700 tx=0x7f17ec009d80).stop 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 shutdown_connections 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f17e005c2b0 0x7f17e005e770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f80f66e0 0x7f17f81091a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 --2- 172.21.15.46:0/766322255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f80f7980 0x7f17f8109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 >> 172.21.15.46:0/766322255 conn(0x7f17f80f1e40 msgr2=0x7f17f806a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:11:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 shutdown_connections 2024-01-21T08:11:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:02.795+0000 7f17fc8ec700 1 -- 172.21.15.46:0/766322255 wait complete. 2024-01-21T08:11:03.127 DEBUG:teuthology.orchestra.run.smithi046:osd.0> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.0.service 2024-01-21T08:11:03.130 INFO:tasks.cephadm:Deploying osd.1 on smithi046 with /dev/vg_nvme/lv_3... 2024-01-21T08:11:03.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-21T08:11:03.161 INFO:journalctl@ceph.osd.0.smithi046.stdout:-- Logs begin at Sun 2024-01-21 07:53:41 UTC. -- 2024-01-21T08:11:03.455 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:11:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0[47710]: debug 2024-01-21T08:11:03.245+0000 7fc098b1b080 -1 Falling back to public interface 2024-01-21T08:11:04.100 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:02.788385+0000 mon.smithi046 (mon.0) 297 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:04.100 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:02.793082+0000 mon.smithi046 (mon.0) 298 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:04.100 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:03.323628+0000 mgr.smithi046.lmxnwc (mgr.14164) 187 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:04.241 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:02.788385+0000 mon.smithi046 (mon.0) 297 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:04.242 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:02.793082+0000 mon.smithi046 (mon.0) 298 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:04.242 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:03.323628+0000 mgr.smithi046.lmxnwc (mgr.14164) 187 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-21T08:11:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-21T08:11:04.684 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-01-21T08:11:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-01-21T08:11:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110836 s, 946 MB/s 2024-01-21T08:11:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-01-21T08:11:05.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2024-01-21T08:11:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:05.323854+0000 mgr.smithi046.lmxnwc (mgr.14164) 188 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:05.323854+0000 mgr.smithi046.lmxnwc (mgr.14164) 188 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:06.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 -- 172.21.15.46:0/667344997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 msgr2=0x7f5e400f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 --2- 172.21.15.46:0/667344997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f6cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f5e3c004660 tx=0x7f5e3c009b30).stop 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 -- 172.21.15.46:0/667344997 shutdown_connections 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 --2- 172.21.15.46:0/667344997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f6cc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 --2- 172.21.15.46:0/667344997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e400f55c0 0x7f5e400f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 -- 172.21.15.46:0/667344997 >> 172.21.15.46:0/667344997 conn(0x7f5e400f0d40 msgr2=0x7f5e400f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 -- 172.21.15.46:0/667344997 shutdown_connections 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.289+0000 7f5e48699700 1 -- 172.21.15.46:0/667344997 wait complete. 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e48699700 1 Processor -- start 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e48699700 1 -- start start 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e48699700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e400f55c0 0x7f5e400f9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e48699700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e48699700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400f8100 con 0x7f5e400f6860 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e48699700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400f8240 con 0x7f5e400f55c0 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e45c34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e45c34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51626/0 (socket says 172.21.15.46:51626) 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e46435700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e400f55c0 0x7f5e400f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e45c34700 1 -- 172.21.15.46:0/3640847809 learned_addr learned my addr 172.21.15.46:0/3640847809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e45c34700 1 -- 172.21.15.46:0/3640847809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e400f55c0 msgr2=0x7f5e400f9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e45c34700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e400f55c0 0x7f5e400f9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.290+0000 7f5e45c34700 1 -- 172.21.15.46:0/3640847809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c005040 con 0x7f5e400f6860 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.291+0000 7f5e45c34700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f7ba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f5e3c000c00 tx=0x7f5e3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.291+0000 7f5e36ffd700 1 -- 172.21.15.46:0/3640847809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e3c019070 con 0x7f5e400f6860 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.291+0000 7f5e36ffd700 1 -- 172.21.15.46:0/3640847809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e3c007e70 con 0x7f5e400f6860 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.291+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e400f84c0 con 0x7f5e400f6860 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.291+0000 7f5e36ffd700 1 -- 172.21.15.46:0/3640847809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e3c014c90 con 0x7f5e400f6860 2024-01-21T08:11:06.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.291+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e400f8a20 con 0x7f5e400f6860 2024-01-21T08:11:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.292+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e40045ad0 con 0x7f5e400f6860 2024-01-21T08:11:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.292+0000 7f5e36ffd700 1 -- 172.21.15.46:0/3640847809 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5e3c025070 con 0x7f5e400f6860 2024-01-21T08:11:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.293+0000 7f5e36ffd700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5e2c05c2b0 0x7f5e2c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.293+0000 7f5e36ffd700 1 -- 172.21.15.46:0/3640847809 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1385+0+0 (secure 0 0 0) 0x7f5e3c075eb0 con 0x7f5e400f6860 2024-01-21T08:11:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.293+0000 7f5e46435700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5e2c05c2b0 0x7f5e2c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.293+0000 7f5e46435700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5e2c05c2b0 0x7f5e2c05e770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5e400f94b0 tx=0x7f5e3000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:11:06.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.297+0000 7f5e36ffd700 1 -- 172.21.15.46:0/3640847809 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e3c049b90 con 0x7f5e400f6860 2024-01-21T08:11:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:06.485+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e400faca0 con 0x7f5e2c05c2b0 2024-01-21T08:11:06.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:06.488321+0000 mon.smithi046 (mon.0) 299 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:11:06.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:06.489579+0000 mon.smithi046 (mon.0) 300 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:11:06.878 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:06.490102+0000 mon.smithi046 (mon.0) 301 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:06.879 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:11:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0[47710]: debug 2024-01-21T08:11:06.724+0000 7fc098b1b080 -1 osd.0 0 log_to_monitors {default=true} 2024-01-21T08:11:06.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:06.488321+0000 mon.smithi046 (mon.0) 299 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:11:06.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:06.489579+0000 mon.smithi046 (mon.0) 300 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:11:06.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:06.490102+0000 mon.smithi046 (mon.0) 301 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:07.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:06.487551+0000 mgr.smithi046.lmxnwc (mgr.14164) 189 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:11:06.489382+0000 mgr.smithi046.lmxnwc (mgr.14164) 190 : cephadm [INF] Applying service osd.None on host smithi046... 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:06.729724+0000 mon.smithi078 (mon.1) 2 : audit [INF] from='osd.0 [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:06.730197+0000 mon.smithi046 (mon.0) 302 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:07.324104+0000 mgr.smithi046.lmxnwc (mgr.14164) 191 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:06.487551+0000 mgr.smithi046.lmxnwc (mgr.14164) 189 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:11:06.489382+0000 mgr.smithi046.lmxnwc (mgr.14164) 190 : cephadm [INF] Applying service osd.None on host smithi046... 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:06.729724+0000 mon.smithi078 (mon.1) 2 : audit [INF] from='osd.0 [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-21T08:11:07.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:06.730197+0000 mon.smithi046 (mon.0) 302 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-21T08:11:07.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:07.324104+0000 mgr.smithi046.lmxnwc (mgr.14164) 191 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T08:11:08.740 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0[47710]: debug 2024-01-21T08:11:08.317+0000 7fc08f08c700 -1 osd.0 0 waiting for initial osdmap 2024-01-21T08:11:08.740 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0[47710]: debug 2024-01-21T08:11:08.320+0000 7fc089804700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:07.546688+0000 mon.smithi046 (mon.0) 303 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:07.546739+0000 mon.smithi046 (mon.0) 304 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:07.546815+0000 mon.smithi046 (mon.0) 305 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:07.547736+0000 mon.smithi078 (mon.1) 3 : audit [INF] from='osd.0 [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:07.548107+0000 mon.smithi046 (mon.0) 306 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:08.311635+0000 mon.smithi046 (mon.0) 307 : audit [INF] from='client.? 172.21.15.46:0/4026414933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82fe182c-88a4-4c1a-a563-a1f281f03bd4"}]: dispatch 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:08.315252+0000 mon.smithi046 (mon.0) 308 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:08.315309+0000 mon.smithi046 (mon.0) 309 : audit [INF] from='client.? 172.21.15.46:0/4026414933' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82fe182c-88a4-4c1a-a563-a1f281f03bd4"}]': finished 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:08.315347+0000 mon.smithi046 (mon.0) 310 : cluster [DBG] osdmap e8: 2 total, 0 up, 2 in 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:08.315438+0000 mon.smithi046 (mon.0) 311 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:08.315806+0000 mon.smithi046 (mon.0) 312 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:08.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:08.318981+0000 mon.smithi046 (mon.0) 313 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:07.546688+0000 mon.smithi046 (mon.0) 303 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:07.546739+0000 mon.smithi046 (mon.0) 304 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:07.546815+0000 mon.smithi046 (mon.0) 305 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:07.547736+0000 mon.smithi078 (mon.1) 3 : audit [INF] from='osd.0 [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:07.548107+0000 mon.smithi046 (mon.0) 306 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:08.311635+0000 mon.smithi046 (mon.0) 307 : audit [INF] from='client.? 172.21.15.46:0/4026414933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82fe182c-88a4-4c1a-a563-a1f281f03bd4"}]: dispatch 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:08.315252+0000 mon.smithi046 (mon.0) 308 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:08.315309+0000 mon.smithi046 (mon.0) 309 : audit [INF] from='client.? 172.21.15.46:0/4026414933' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82fe182c-88a4-4c1a-a563-a1f281f03bd4"}]': finished 2024-01-21T08:11:08.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:08.315347+0000 mon.smithi046 (mon.0) 310 : cluster [DBG] osdmap e8: 2 total, 0 up, 2 in 2024-01-21T08:11:08.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:08.315438+0000 mon.smithi046 (mon.0) 311 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:08.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:08.315806+0000 mon.smithi046 (mon.0) 312 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:08.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:08.318981+0000 mon.smithi046 (mon.0) 313 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:09.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:08.741190+0000 mon.smithi046 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.46:0/2138823857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:11:09.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:09.316628+0000 mon.smithi046 (mon.0) 315 : cluster [INF] osd.0 [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] boot 2024-01-21T08:11:09.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:09.316687+0000 mon.smithi046 (mon.0) 316 : cluster [DBG] osdmap e9: 2 total, 1 up, 2 in 2024-01-21T08:11:09.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:09.316850+0000 mon.smithi046 (mon.0) 317 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:09.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:09.317234+0000 mon.smithi046 (mon.0) 318 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:09.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:09.324430+0000 mgr.smithi046.lmxnwc (mgr.14164) 192 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:09.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:08.741190+0000 mon.smithi046 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.46:0/2138823857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:11:09.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:09.316628+0000 mon.smithi046 (mon.0) 315 : cluster [INF] osd.0 [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] boot 2024-01-21T08:11:09.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:09.316687+0000 mon.smithi046 (mon.0) 316 : cluster [DBG] osdmap e9: 2 total, 1 up, 2 in 2024-01-21T08:11:09.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:09.316850+0000 mon.smithi046 (mon.0) 317 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-21T08:11:09.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:09.317234+0000 mon.smithi046 (mon.0) 318 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:09.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:09.324430+0000 mgr.smithi046.lmxnwc (mgr.14164) 192 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:10.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:07.734687+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:11:10.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:07.734784+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:11:10.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:07.734687+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:11:10.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:07.734784+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:11:11.904 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:10.552137+0000 mon.smithi046 (mon.0) 319 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-01-21T08:11:11.904 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:10.552521+0000 mon.smithi046 (mon.0) 320 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:11.904 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:11.324911+0000 mgr.smithi046.lmxnwc (mgr.14164) 193 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:11.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:10.552137+0000 mon.smithi046 (mon.0) 319 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-01-21T08:11:11.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:10.552521+0000 mon.smithi046 (mon.0) 320 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:11.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:11.324911+0000 mgr.smithi046.lmxnwc (mgr.14164) 193 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:13.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:13.325209+0000 mgr.smithi046.lmxnwc (mgr.14164) 194 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:13.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:13.325209+0000 mgr.smithi046.lmxnwc (mgr.14164) 194 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:14.604 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:13.941714+0000 mon.smithi046 (mon.0) 321 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-21T08:11:14.604 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:13.942342+0000 mon.smithi046 (mon.0) 322 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:14.605 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:11:13.942831+0000 mgr.smithi046.lmxnwc (mgr.14164) 195 : cephadm [INF] Deploying daemon osd.1 on smithi046 2024-01-21T08:11:14.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:14 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:13.941714+0000 mon.smithi046 (mon.0) 321 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-21T08:11:14.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:14 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:13.942342+0000 mon.smithi046 (mon.0) 322 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:14.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:14 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:11:13.942831+0000 mgr.smithi046.lmxnwc (mgr.14164) 195 : cephadm [INF] Deploying daemon osd.1 on smithi046 2024-01-21T08:11:15.608 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:15.325462+0000 mgr.smithi046.lmxnwc (mgr.14164) 196 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:15.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:15.325462+0000 mgr.smithi046.lmxnwc (mgr.14164) 196 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:16.277 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 1 on host 'smithi046' 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.273+0000 7f5e36ffd700 1 -- 172.21.15.46:0/3640847809 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5e400faca0 con 0x7f5e2c05c2b0 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5e2c05c2b0 msgr2=0x7f5e2c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5e2c05c2b0 0x7f5e2c05e770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5e400f94b0 tx=0x7f5e3000b040).stop 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 msgr2=0x7f5e400f7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f7ba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f5e3c000c00 tx=0x7f5e3c00b040).stop 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 shutdown_connections 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5e2c05c2b0 0x7f5e2c05e770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e400f55c0 0x7f5e400f9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 --2- 172.21.15.46:0/3640847809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400f6860 0x7f5e400f7ba0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 >> 172.21.15.46:0/3640847809 conn(0x7f5e400f0d40 msgr2=0x7f5e4005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:11:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 shutdown_connections 2024-01-21T08:11:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:16.275+0000 7f5e48699700 1 -- 172.21.15.46:0/3640847809 wait complete. 2024-01-21T08:11:16.616 DEBUG:teuthology.orchestra.run.smithi046:osd.1> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.1.service 2024-01-21T08:11:16.619 INFO:tasks.cephadm:Deploying osd.2 on smithi046 with /dev/vg_nvme/lv_2... 2024-01-21T08:11:16.619 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-21T08:11:16.649 INFO:journalctl@ceph.osd.1.smithi046.stdout:-- Logs begin at Sun 2024-01-21 07:53:41 UTC. -- 2024-01-21T08:11:17.130 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:11:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1[50858]: debug 2024-01-21T08:11:16.757+0000 7fa98db58080 -1 Falling back to public interface 2024-01-21T08:11:17.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:16.270571+0000 mon.smithi046 (mon.0) 323 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:17.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:16.273978+0000 mon.smithi046 (mon.0) 324 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:17.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:16.270571+0000 mon.smithi046 (mon.0) 323 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:17.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:16.273978+0000 mon.smithi046 (mon.0) 324 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:18.076 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-21T08:11:18.076 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-21T08:11:18.077 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-01-21T08:11:18.077 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-01-21T08:11:18.077 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00930342 s, 1.1 GB/s 2024-01-21T08:11:18.077 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-01-21T08:11:18.374 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:17.325809+0000 mgr.smithi046.lmxnwc (mgr.14164) 197 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:18.458 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2024-01-21T08:11:18.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:17.325809+0000 mgr.smithi046.lmxnwc (mgr.14164) 197 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:19.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 -- 172.21.15.46:0/1876587582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc00f9820 msgr2=0x7f8dc00fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:19.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/1876587582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc00f9820 0x7f8dc00fbc70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f8dac004660 tx=0x7f8dac009b30).stop 2024-01-21T08:11:19.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 -- 172.21.15.46:0/1876587582 shutdown_connections 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/1876587582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc00f9820 0x7f8dc00fbc70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/1876587582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc0060310 0x7f8dc00f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 -- 172.21.15.46:0/1876587582 >> 172.21.15.46:0/1876587582 conn(0x7f8dc00f0d40 msgr2=0x7f8dc00f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 -- 172.21.15.46:0/1876587582 shutdown_connections 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.565+0000 7f8dc6b67700 1 -- 172.21.15.46:0/1876587582 wait complete. 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.566+0000 7f8dc6b67700 1 Processor -- start 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.566+0000 7f8dc6b67700 1 -- start start 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.566+0000 7f8dc6b67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc0060310 0x7f8dc00f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.566+0000 7f8dc6b67700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc00f9820 0x7f8dc00f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.566+0000 7f8dc6b67700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00f5700 con 0x7f8dc0060310 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc6b67700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00f5840 con 0x7f8dc00f9820 2024-01-21T08:11:19.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc4903700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc0060310 0x7f8dc00f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc4903700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc0060310 0x7f8dc00f6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44172/0 (socket says 172.21.15.46:44172) 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc4903700 1 -- 172.21.15.46:0/2079312039 learned_addr learned my addr 172.21.15.46:0/2079312039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dbffff700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc00f9820 0x7f8dc00f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc4903700 1 -- 172.21.15.46:0/2079312039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc00f9820 msgr2=0x7f8dc00f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc4903700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc00f9820 0x7f8dc00f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc4903700 1 -- 172.21.15.46:0/2079312039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac005040 con 0x7f8dc0060310 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc4903700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc0060310 0x7f8dc00f6d20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f8db400e7f0 tx=0x7f8db400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dbd7fa700 1 -- 172.21.15.46:0/2079312039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8db40092e0 con 0x7f8dc0060310 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dbd7fa700 1 -- 172.21.15.46:0/2079312039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8db400f040 con 0x7f8dc0060310 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc00f5b20 con 0x7f8dc0060310 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dbd7fa700 1 -- 172.21.15.46:0/2079312039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8db40146c0 con 0x7f8dc0060310 2024-01-21T08:11:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.567+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc010ef10 con 0x7f8dc0060310 2024-01-21T08:11:19.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.569+0000 7f8dbd7fa700 1 -- 172.21.15.46:0/2079312039 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8db401b030 con 0x7f8dc0060310 2024-01-21T08:11:19.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.569+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc0045ad0 con 0x7f8dc0060310 2024-01-21T08:11:19.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.569+0000 7f8dbd7fa700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8db005c2b0 0x7f8db005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.569+0000 7f8dbd7fa700 1 -- 172.21.15.46:0/2079312039 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1997+0+0 (secure 0 0 0) 0x7f8db4074f30 con 0x7f8dc0060310 2024-01-21T08:11:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.569+0000 7f8dbffff700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8db005c2b0 0x7f8db005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.569+0000 7f8dbffff700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8db005c2b0 0x7f8db005e770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f8dac005010 tx=0x7f8dac00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:11:19.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.577+0000 7f8dbd7fa700 1 -- 172.21.15.46:0/2079312039 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8db40a12b0 con 0x7f8dc0060310 2024-01-21T08:11:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:19.761+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dc00ff000 con 0x7f8db005c2b0 2024-01-21T08:11:20.314 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:11:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1[50858]: debug 2024-01-21T08:11:20.283+0000 7fa98db58080 -1 osd.1 0 log_to_monitors {default=true} 2024-01-21T08:11:20.695 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:19.326033+0000 mgr.smithi046.lmxnwc (mgr.14164) 198 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:20.695 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:19.763564+0000 mon.smithi046 (mon.0) 325 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:11:20.695 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:19.764860+0000 mon.smithi046 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:11:20.695 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:19.765446+0000 mon.smithi046 (mon.0) 327 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:20.696 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:20.286523+0000 mon.smithi046 (mon.0) 328 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-21T08:11:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:19.326033+0000 mgr.smithi046.lmxnwc (mgr.14164) 198 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:20.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:19.763564+0000 mon.smithi046 (mon.0) 325 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:11:20.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:19.764860+0000 mon.smithi046 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:11:20.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:19.765446+0000 mon.smithi046 (mon.0) 327 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:20.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:20.286523+0000 mon.smithi046 (mon.0) 328 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:19.762790+0000 mgr.smithi046.lmxnwc (mgr.14164) 199 : audit [DBG] from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:11:19.764652+0000 mgr.smithi046.lmxnwc (mgr.14164) 200 : cephadm [INF] Applying service osd.None on host smithi046... 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:20.315784+0000 mon.smithi046 (mon.0) 329 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:20.315826+0000 mon.smithi046 (mon.0) 330 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:20.315921+0000 mon.smithi046 (mon.0) 331 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:20.316266+0000 mon.smithi046 (mon.0) 332 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:20.744280+0000 mon.smithi046 (mon.0) 333 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:11:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:20.748014+0000 mon.smithi046 (mon.0) 334 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:11:21.742 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1[50858]: debug 2024-01-21T08:11:21.318+0000 7fa9840c9700 -1 osd.1 0 waiting for initial osdmap 2024-01-21T08:11:21.742 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:11:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1[50858]: debug 2024-01-21T08:11:21.323+0000 7fa97e040700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:19.762790+0000 mgr.smithi046.lmxnwc (mgr.14164) 199 : audit [DBG] from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:11:19.764652+0000 mgr.smithi046.lmxnwc (mgr.14164) 200 : cephadm [INF] Applying service osd.None on host smithi046... 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:20.315784+0000 mon.smithi046 (mon.0) 329 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:20.315826+0000 mon.smithi046 (mon.0) 330 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:20.315921+0000 mon.smithi046 (mon.0) 331 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:20.316266+0000 mon.smithi046 (mon.0) 332 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:20.744280+0000 mon.smithi046 (mon.0) 333 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:11:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:20.748014+0000 mon.smithi046 (mon.0) 334 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:11:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.316271+0000 mon.smithi046 (mon.0) 335 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-01-21T08:11:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:21.316320+0000 mon.smithi046 (mon.0) 336 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-01-21T08:11:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.316644+0000 mon.smithi046 (mon.0) 337 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.319543+0000 mon.smithi046 (mon.0) 338 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:21.326291+0000 mgr.smithi046.lmxnwc (mgr.14164) 201 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.372214+0000 mon.smithi046 (mon.0) 339 : audit [INF] from='client.? 172.21.15.46:0/531966123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4715fc0d-a749-4eab-ad3e-1934b586b33d"}]: dispatch 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:21.375539+0000 mon.smithi046 (mon.0) 340 : cluster [INF] osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] boot 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.375585+0000 mon.smithi046 (mon.0) 341 : audit [INF] from='client.? 172.21.15.46:0/531966123' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4715fc0d-a749-4eab-ad3e-1934b586b33d"}]': finished 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:21.375625+0000 mon.smithi046 (mon.0) 342 : cluster [DBG] osdmap e13: 3 total, 2 up, 3 in 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.375867+0000 mon.smithi046 (mon.0) 343 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.376056+0000 mon.smithi046 (mon.0) 344 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:21.793663+0000 mon.smithi046 (mon.0) 345 : audit [DBG] from='client.? 172.21.15.46:0/1176134083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:11:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.316271+0000 mon.smithi046 (mon.0) 335 : audit [INF] from='osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:21.316320+0000 mon.smithi046 (mon.0) 336 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.316644+0000 mon.smithi046 (mon.0) 337 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.319543+0000 mon.smithi046 (mon.0) 338 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:21.326291+0000 mgr.smithi046.lmxnwc (mgr.14164) 201 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.372214+0000 mon.smithi046 (mon.0) 339 : audit [INF] from='client.? 172.21.15.46:0/531966123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4715fc0d-a749-4eab-ad3e-1934b586b33d"}]: dispatch 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:21.375539+0000 mon.smithi046 (mon.0) 340 : cluster [INF] osd.1 [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] boot 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.375585+0000 mon.smithi046 (mon.0) 341 : audit [INF] from='client.? 172.21.15.46:0/531966123' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4715fc0d-a749-4eab-ad3e-1934b586b33d"}]': finished 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:21.375625+0000 mon.smithi046 (mon.0) 342 : cluster [DBG] osdmap e13: 3 total, 2 up, 3 in 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.375867+0000 mon.smithi046 (mon.0) 343 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.376056+0000 mon.smithi046 (mon.0) 344 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:22.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:21.793663+0000 mon.smithi046 (mon.0) 345 : audit [DBG] from='client.? 172.21.15.46:0/1176134083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:11:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:21.286452+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:11:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:21.286556+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:11:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:21.286452+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:11:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:21.286556+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:11:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:23.326543+0000 mgr.smithi046.lmxnwc (mgr.14164) 202 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:24.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:23.328422+0000 mon.smithi046 (mon.0) 346 : cluster [DBG] osdmap e14: 3 total, 2 up, 3 in 2024-01-21T08:11:24.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:23.328813+0000 mon.smithi046 (mon.0) 347 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:24.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:23.326543+0000 mgr.smithi046.lmxnwc (mgr.14164) 202 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:24.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:23.328422+0000 mon.smithi046 (mon.0) 346 : cluster [DBG] osdmap e14: 3 total, 2 up, 3 in 2024-01-21T08:11:24.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:23.328813+0000 mon.smithi046 (mon.0) 347 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:26.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:25.326868+0000 mgr.smithi046.lmxnwc (mgr.14164) 203 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:26.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:25.326868+0000 mgr.smithi046.lmxnwc (mgr.14164) 203 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:27.603 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:26.957012+0000 mon.smithi046 (mon.0) 348 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-21T08:11:27.603 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:26.957650+0000 mon.smithi046 (mon.0) 349 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:26.957012+0000 mon.smithi046 (mon.0) 348 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-21T08:11:27.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:26.957650+0000 mon.smithi046 (mon.0) 349 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:28.626 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:11:26.958175+0000 mgr.smithi046.lmxnwc (mgr.14164) 204 : cephadm [INF] Deploying daemon osd.2 on smithi046 2024-01-21T08:11:28.626 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:27.327142+0000 mgr.smithi046.lmxnwc (mgr.14164) 205 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:28.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:11:26.958175+0000 mgr.smithi046.lmxnwc (mgr.14164) 204 : cephadm [INF] Deploying daemon osd.2 on smithi046 2024-01-21T08:11:28.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:27.327142+0000 mgr.smithi046.lmxnwc (mgr.14164) 205 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:29.327 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 2 on host 'smithi046' 2024-01-21T08:11:29.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.323+0000 7f8dbd7fa700 1 -- 172.21.15.46:0/2079312039 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8dac01a040 con 0x7f8db005c2b0 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8db005c2b0 msgr2=0x7f8db005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8db005c2b0 0x7f8db005e770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f8dac005010 tx=0x7f8dac00b040).stop 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc0060310 msgr2=0x7f8dc00f6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc0060310 0x7f8dc00f6d20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f8db400e7f0 tx=0x7f8db400b040).stop 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 shutdown_connections 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8db005c2b0 0x7f8db005e770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dc0060310 0x7f8dc00f6d20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 --2- 172.21.15.46:0/2079312039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc00f9820 0x7f8dc00f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 >> 172.21.15.46:0/2079312039 conn(0x7f8dc00f0d40 msgr2=0x7f8dc00f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:11:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 shutdown_connections 2024-01-21T08:11:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:11:29.326+0000 7f8dc6b67700 1 -- 172.21.15.46:0/2079312039 wait complete. 2024-01-21T08:11:29.708 DEBUG:teuthology.orchestra.run.smithi046:osd.2> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.2.service 2024-01-21T08:11:29.711 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_4... 2024-01-21T08:11:29.711 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-21T08:11:29.742 INFO:journalctl@ceph.osd.2.smithi046.stdout:-- Logs begin at Sun 2024-01-21 07:53:41 UTC. -- 2024-01-21T08:11:29.990 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:11:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2[54012]: debug 2024-01-21T08:11:29.774+0000 7f96f4a5b080 -1 Falling back to public interface 2024-01-21T08:11:30.609 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:29.320313+0000 mon.smithi046 (mon.0) 350 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:30.609 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:29.323798+0000 mon.smithi046 (mon.0) 351 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:30.609 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:29.327424+0000 mgr.smithi046.lmxnwc (mgr.14164) 206 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:29.320313+0000 mon.smithi046 (mon.0) 350 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:29.323798+0000 mon.smithi046 (mon.0) 351 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:11:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:29.327424+0000 mgr.smithi046.lmxnwc (mgr.14164) 206 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:31.201 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-21T08:11:31.201 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-21T08:11:31.201 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-01-21T08:11:31.201 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-01-21T08:11:31.201 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148504 s, 706 MB/s 2024-01-21T08:11:31.201 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-01-21T08:11:31.592 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:31.327803+0000 mgr.smithi046.lmxnwc (mgr.14164) 207 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:31.611 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2024-01-21T08:11:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:31.327803+0000 mgr.smithi046.lmxnwc (mgr.14164) 207 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 -- 172.21.15.78:0/2177127210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 msgr2=0x7fa2440f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 --2- 172.21.15.78:0/2177127210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2440f6cc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fa240004660 tx=0x7fa240009b30).stop 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 -- 172.21.15.78:0/2177127210 shutdown_connections 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 --2- 172.21.15.78:0/2177127210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2440f6cc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 --2- 172.21.15.78:0/2177127210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2440f55c0 0x7fa2440f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 -- 172.21.15.78:0/2177127210 >> 172.21.15.78:0/2177127210 conn(0x7fa2440f0d40 msgr2=0x7fa2440f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 -- 172.21.15.78:0/2177127210 shutdown_connections 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.880+0000 7fa24c533700 1 -- 172.21.15.78:0/2177127210 wait complete. 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24c533700 1 Processor -- start 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24c533700 1 -- start start 2024-01-21T08:11:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24c533700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2440f55c0 0x7fa244105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24c533700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2441063c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24c533700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2441069c0 con 0x7fa2440f55c0 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24c533700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2441071d0 con 0x7fa2440f6860 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa249ace700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2441063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa249ace700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2441063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57484/0 (socket says 172.21.15.78:57484) 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa249ace700 1 -- 172.21.15.78:0/1220333115 learned_addr learned my addr 172.21.15.78:0/1220333115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24a2cf700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2440f55c0 0x7fa244105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa249ace700 1 -- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2440f55c0 msgr2=0x7fa244105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa249ace700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2440f55c0 0x7fa244105ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa249ace700 1 -- 172.21.15.78:0/1220333115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240005040 con 0x7fa2440f6860 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa249ace700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2441063c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa240009f50 tx=0x7fa240009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:11:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa240018070 con 0x7fa2440f6860 2024-01-21T08:11:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa240013410 con 0x7fa2440f6860 2024-01-21T08:11:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2441073f0 con 0x7fa2440f6860 2024-01-21T08:11:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.881+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24001c3c0 con 0x7fa2440f6860 2024-01-21T08:11:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.882+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa244107950 con 0x7fa2440f6860 2024-01-21T08:11:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.882+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2440f49b0 con 0x7fa2440f6860 2024-01-21T08:11:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.883+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa24001d420 con 0x7fa2440f6860 2024-01-21T08:11:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.883+0000 7fa23affd700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa23005c2b0 0x7fa23005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:11:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.883+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2429+0+0 (secure 0 0 0) 0x7fa240076ee0 con 0x7fa2440f6860 2024-01-21T08:11:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.883+0000 7fa24a2cf700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa23005c2b0 0x7fa23005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:11:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.884+0000 7fa24a2cf700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa23005c2b0 0x7fa23005e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fa244105c40 tx=0x7fa23400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:11:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:32.888+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa24004a7b0 con 0x7fa2440f6860 2024-01-21T08:11:33.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:11:33.060+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2440fdee0 con 0x7fa23005c2b0 2024-01-21T08:11:33.394 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:33.064154+0000 mon.smithi046 (mon.0) 352 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:11:33.394 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:33.065945+0000 mon.smithi046 (mon.0) 353 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:11:33.394 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:33.066693+0000 mon.smithi046 (mon.0) 354 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:33.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:11:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2[54012]: debug 2024-01-21T08:11:33.212+0000 7f96f4a5b080 -1 osd.2 0 log_to_monitors {default=true} 2024-01-21T08:11:33.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:33.064154+0000 mon.smithi046 (mon.0) 352 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:11:33.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:33.065945+0000 mon.smithi046 (mon.0) 353 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:11:33.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:33.066693+0000 mon.smithi046 (mon.0) 354 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:11:34.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:33.063198+0000 mgr.smithi046.lmxnwc (mgr.14164) 208 : audit [DBG] from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:11:34.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:11:33.065682+0000 mgr.smithi046.lmxnwc (mgr.14164) 209 : cephadm [INF] Applying service osd.None on host smithi078... 2024-01-21T08:11:34.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:33.216608+0000 mon.smithi046 (mon.0) 355 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-21T08:11:34.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:33.328268+0000 mgr.smithi046.lmxnwc (mgr.14164) 210 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:34.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:34 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:33.063198+0000 mgr.smithi046.lmxnwc (mgr.14164) 208 : audit [DBG] from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:11:34.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:34 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:11:33.065682+0000 mgr.smithi046.lmxnwc (mgr.14164) 209 : cephadm [INF] Applying service osd.None on host smithi078... 2024-01-21T08:11:34.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:34 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:33.216608+0000 mon.smithi046 (mon.0) 355 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-21T08:11:34.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:34 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:33.328268+0000 mgr.smithi046.lmxnwc (mgr.14164) 210 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:35.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:11:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2[54012]: debug 2024-01-21T08:11:35.128+0000 7f96eafcc700 -1 osd.2 0 waiting for initial osdmap 2024-01-21T08:11:35.490 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:11:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2[54012]: debug 2024-01-21T08:11:35.136+0000 7f96e3f41700 -1 osd.2 16 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-01-21T08:11:35.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:34.123376+0000 mon.smithi046 (mon.0) 356 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-21T08:11:35.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:34.123460+0000 mon.smithi046 (mon.0) 357 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-01-21T08:11:35.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:34.123752+0000 mon.smithi046 (mon.0) 358 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:35.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:34.124209+0000 mon.smithi046 (mon.0) 359 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:35.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:34.123376+0000 mon.smithi046 (mon.0) 356 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-21T08:11:35.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:34.123460+0000 mon.smithi046 (mon.0) 357 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-01-21T08:11:35.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:34.123752+0000 mon.smithi046 (mon.0) 358 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:35.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:34.124209+0000 mon.smithi046 (mon.0) 359 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-01-21T08:11:36.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:35.125448+0000 mon.smithi046 (mon.0) 360 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-01-21T08:11:36.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:35.125544+0000 mon.smithi046 (mon.0) 361 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-01-21T08:11:36.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:35.125960+0000 mon.smithi046 (mon.0) 362 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:36.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:35.131984+0000 mon.smithi046 (mon.0) 363 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:36.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:35.328800+0000 mgr.smithi046.lmxnwc (mgr.14164) 211 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:36.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:36 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:35.125448+0000 mon.smithi046 (mon.0) 360 : audit [INF] from='osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-01-21T08:11:36.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:36 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:35.125544+0000 mon.smithi046 (mon.0) 361 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-01-21T08:11:36.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:36 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:35.125960+0000 mon.smithi046 (mon.0) 362 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:36.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:36 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:35.131984+0000 mon.smithi046 (mon.0) 363 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:36.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:36 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:35.328800+0000 mgr.smithi046.lmxnwc (mgr.14164) 211 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-01-21T08:11:37.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:34.210765+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:11:37.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:34.210886+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:11:37.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:36.129218+0000 mon.smithi046 (mon.0) 364 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:37.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:36.133440+0000 mon.smithi046 (mon.0) 365 : cluster [INF] osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] boot 2024-01-21T08:11:37.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:36.133471+0000 mon.smithi046 (mon.0) 366 : cluster [DBG] osdmap e17: 3 total, 3 up, 3 in 2024-01-21T08:11:37.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:11:36.134059+0000 mon.smithi046 (mon.0) 367 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:37.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:34.210765+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:11:37.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:34.210886+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:11:37.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:36.129218+0000 mon.smithi046 (mon.0) 364 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:37.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:36.133440+0000 mon.smithi046 (mon.0) 365 : cluster [INF] osd.2 [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] boot 2024-01-21T08:11:37.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:36.133471+0000 mon.smithi046 (mon.0) 366 : cluster [DBG] osdmap e17: 3 total, 3 up, 3 in 2024-01-21T08:11:37.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:11:36.134059+0000 mon.smithi046 (mon.0) 367 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-21T08:11:38.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:37.144314+0000 mon.smithi046 (mon.0) 368 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-01-21T08:11:38.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:37.329278+0000 mgr.smithi046.lmxnwc (mgr.14164) 212 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:38.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:38 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:37.144314+0000 mon.smithi046 (mon.0) 368 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-01-21T08:11:38.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:38 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:37.329278+0000 mgr.smithi046.lmxnwc (mgr.14164) 212 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:39.329582+0000 mgr.smithi046.lmxnwc (mgr.14164) 213 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:40.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:39.329582+0000 mgr.smithi046.lmxnwc (mgr.14164) 213 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:41.329873+0000 mgr.smithi046.lmxnwc (mgr.14164) 214 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:41.329873+0000 mgr.smithi046.lmxnwc (mgr.14164) 214 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:43.330209+0000 mgr.smithi046.lmxnwc (mgr.14164) 215 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:43.330209+0000 mgr.smithi046.lmxnwc (mgr.14164) 215 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:45.330563+0000 mgr.smithi046.lmxnwc (mgr.14164) 216 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:45.330563+0000 mgr.smithi046.lmxnwc (mgr.14164) 216 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:47.330994+0000 mgr.smithi046.lmxnwc (mgr.14164) 217 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:47.330994+0000 mgr.smithi046.lmxnwc (mgr.14164) 217 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:49.331415+0000 mgr.smithi046.lmxnwc (mgr.14164) 218 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:49.331415+0000 mgr.smithi046.lmxnwc (mgr.14164) 218 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:51.331721+0000 mgr.smithi046.lmxnwc (mgr.14164) 219 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:51.331721+0000 mgr.smithi046.lmxnwc (mgr.14164) 219 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:53.332167+0000 mgr.smithi046.lmxnwc (mgr.14164) 220 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:53.332167+0000 mgr.smithi046.lmxnwc (mgr.14164) 220 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:55.332546+0000 mgr.smithi046.lmxnwc (mgr.14164) 221 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:55.332546+0000 mgr.smithi046.lmxnwc (mgr.14164) 221 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:11:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:57.332985+0000 mgr.smithi046.lmxnwc (mgr.14164) 222 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:11:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:11:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:57.332985+0000 mgr.smithi046.lmxnwc (mgr.14164) 222 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:11:59.333420+0000 mgr.smithi046.lmxnwc (mgr.14164) 223 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:11:59.333420+0000 mgr.smithi046.lmxnwc (mgr.14164) 223 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:01.333845+0000 mgr.smithi046.lmxnwc (mgr.14164) 224 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:01.333845+0000 mgr.smithi046.lmxnwc (mgr.14164) 224 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:03.334327+0000 mgr.smithi046.lmxnwc (mgr.14164) 225 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:03.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:03.334327+0000 mgr.smithi046.lmxnwc (mgr.14164) 225 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:05.334758+0000 mgr.smithi046.lmxnwc (mgr.14164) 226 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:05.334758+0000 mgr.smithi046.lmxnwc (mgr.14164) 226 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:07.335153+0000 mgr.smithi046.lmxnwc (mgr.14164) 227 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:07.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:07.335153+0000 mgr.smithi046.lmxnwc (mgr.14164) 227 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:09.335543+0000 mgr.smithi046.lmxnwc (mgr.14164) 228 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:10.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:09.335543+0000 mgr.smithi046.lmxnwc (mgr.14164) 228 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:11.335926+0000 mgr.smithi046.lmxnwc (mgr.14164) 229 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:11.335926+0000 mgr.smithi046.lmxnwc (mgr.14164) 229 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:13.336386+0000 mgr.smithi046.lmxnwc (mgr.14164) 230 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:13.336386+0000 mgr.smithi046.lmxnwc (mgr.14164) 230 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:15.336841+0000 mgr.smithi046.lmxnwc (mgr.14164) 231 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:15.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:15.336841+0000 mgr.smithi046.lmxnwc (mgr.14164) 231 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:17.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:17.337186+0000 mgr.smithi046.lmxnwc (mgr.14164) 232 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:17.337186+0000 mgr.smithi046.lmxnwc (mgr.14164) 232 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:19.337592+0000 mgr.smithi046.lmxnwc (mgr.14164) 233 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:19.337592+0000 mgr.smithi046.lmxnwc (mgr.14164) 233 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:12:20.748233+0000 mon.smithi046 (mon.0) 369 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:12:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:12:20.749326+0000 mon.smithi046 (mon.0) 370 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:12:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:12:20.748233+0000 mon.smithi046 (mon.0) 369 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:12:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:12:20.749326+0000 mon.smithi046 (mon.0) 370 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:12:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:21.337859+0000 mgr.smithi046.lmxnwc (mgr.14164) 234 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:21.337859+0000 mgr.smithi046.lmxnwc (mgr.14164) 234 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:23.338271+0000 mgr.smithi046.lmxnwc (mgr.14164) 235 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:23.338271+0000 mgr.smithi046.lmxnwc (mgr.14164) 235 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:25.338672+0000 mgr.smithi046.lmxnwc (mgr.14164) 236 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:25.338672+0000 mgr.smithi046.lmxnwc (mgr.14164) 236 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:27.339089+0000 mgr.smithi046.lmxnwc (mgr.14164) 237 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:27.339089+0000 mgr.smithi046.lmxnwc (mgr.14164) 237 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:29.339513+0000 mgr.smithi046.lmxnwc (mgr.14164) 238 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:30.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:29.339513+0000 mgr.smithi046.lmxnwc (mgr.14164) 238 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:31.339836+0000 mgr.smithi046.lmxnwc (mgr.14164) 239 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:31.339836+0000 mgr.smithi046.lmxnwc (mgr.14164) 239 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:33.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:33.340262+0000 mgr.smithi046.lmxnwc (mgr.14164) 240 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:33.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:33.340262+0000 mgr.smithi046.lmxnwc (mgr.14164) 240 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:35.340629+0000 mgr.smithi046.lmxnwc (mgr.14164) 241 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:35.340629+0000 mgr.smithi046.lmxnwc (mgr.14164) 241 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:37.340990+0000 mgr.smithi046.lmxnwc (mgr.14164) 242 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:37.340990+0000 mgr.smithi046.lmxnwc (mgr.14164) 242 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:39.341355+0000 mgr.smithi046.lmxnwc (mgr.14164) 243 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:40.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:39.341355+0000 mgr.smithi046.lmxnwc (mgr.14164) 243 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:12:40.777396+0000 mgr.smithi046.lmxnwc (mgr.14164) 244 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi078 ... 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Verifying port 9100 ... 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:12:41.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: yield (conn, connr) 2024-01-21T08:12:41.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: code, '\n'.join(err))) 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi078 ... 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Verifying port 9100 ... 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:12:41.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:12:40.780626+0000 mgr.smithi046.lmxnwc (mgr.14164) 245 : cephadm [INF] Deploying daemon prometheus.smithi078 on smithi078 2024-01-21T08:12:41.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:41.341752+0000 mgr.smithi046.lmxnwc (mgr.14164) 246 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:41.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:12:40.777396+0000 mgr.smithi046.lmxnwc (mgr.14164) 244 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi078 ... 2024-01-21T08:12:41.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Verifying port 9100 ... 2024-01-21T08:12:41.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 1 from systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:12:41.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:12:41.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:12:41.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ports=daemon_ports) 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:12:41.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: Failed command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: yield (conn, connr) 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: code, '\n'.join(err))) 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi078 ... 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Verifying port 9100 ... 2024-01-21T08:12:41.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 1 from systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ports=daemon_ports) 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T08:12:41.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T08:12:41.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T08:12:41.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:12:41.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:12:41.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: Failed command: systemctl start ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:12:41.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:12:40.780626+0000 mgr.smithi046.lmxnwc (mgr.14164) 245 : cephadm [INF] Deploying daemon prometheus.smithi078 on smithi078 2024-01-21T08:12:41.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:41.341752+0000 mgr.smithi046.lmxnwc (mgr.14164) 246 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:12:41.597237+0000 mon.smithi078 (mon.1) 4 : audit [INF] from='client.? 172.21.15.78:0/1945694066' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46321964-fb94-4a83-873e-ffe70b11e1c5"}]: dispatch 2024-01-21T08:12:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:12:41.597772+0000 mon.smithi046 (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46321964-fb94-4a83-873e-ffe70b11e1c5"}]: dispatch 2024-01-21T08:12:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:12:41.602026+0000 mon.smithi046 (mon.0) 372 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46321964-fb94-4a83-873e-ffe70b11e1c5"}]': finished 2024-01-21T08:12:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:41.602105+0000 mon.smithi046 (mon.0) 373 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2024-01-21T08:12:42.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:12:41.602230+0000 mon.smithi046 (mon.0) 374 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:12:42.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:12:42.007925+0000 mon.smithi078 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.78:0/631595612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:12:42.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:12:41.597237+0000 mon.smithi078 (mon.1) 4 : audit [INF] from='client.? 172.21.15.78:0/1945694066' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46321964-fb94-4a83-873e-ffe70b11e1c5"}]: dispatch 2024-01-21T08:12:42.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:12:41.597772+0000 mon.smithi046 (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46321964-fb94-4a83-873e-ffe70b11e1c5"}]: dispatch 2024-01-21T08:12:42.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:12:41.602026+0000 mon.smithi046 (mon.0) 372 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46321964-fb94-4a83-873e-ffe70b11e1c5"}]': finished 2024-01-21T08:12:42.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:41.602105+0000 mon.smithi046 (mon.0) 373 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2024-01-21T08:12:42.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:12:41.602230+0000 mon.smithi046 (mon.0) 374 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:12:42.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:12:42.007925+0000 mon.smithi078 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.78:0/631595612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:12:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:43.342183+0000 mgr.smithi046.lmxnwc (mgr.14164) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:43.342183+0000 mgr.smithi046.lmxnwc (mgr.14164) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:45.659 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:45.342578+0000 mgr.smithi046.lmxnwc (mgr.14164) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:45.342578+0000 mgr.smithi046.lmxnwc (mgr.14164) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:47.342894+0000 mgr.smithi046.lmxnwc (mgr.14164) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:47.342894+0000 mgr.smithi046.lmxnwc (mgr.14164) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:49.343237+0000 mgr.smithi046.lmxnwc (mgr.14164) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:49.343237+0000 mgr.smithi046.lmxnwc (mgr.14164) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:51.343674+0000 mgr.smithi046.lmxnwc (mgr.14164) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:51.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:51.343674+0000 mgr.smithi046.lmxnwc (mgr.14164) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:53.344067+0000 mgr.smithi046.lmxnwc (mgr.14164) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:53.344067+0000 mgr.smithi046.lmxnwc (mgr.14164) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:55.344385+0000 mgr.smithi046.lmxnwc (mgr.14164) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:55.344385+0000 mgr.smithi046.lmxnwc (mgr.14164) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:12:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:57.344718+0000 mgr.smithi046.lmxnwc (mgr.14164) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:12:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:12:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:57.344718+0000 mgr.smithi046.lmxnwc (mgr.14164) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:12:59.344967+0000 mgr.smithi046.lmxnwc (mgr.14164) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:12:59.344967+0000 mgr.smithi046.lmxnwc (mgr.14164) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:01.345243+0000 mgr.smithi046.lmxnwc (mgr.14164) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:01.345243+0000 mgr.smithi046.lmxnwc (mgr.14164) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:03.345626+0000 mgr.smithi046.lmxnwc (mgr.14164) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:03.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:03.345626+0000 mgr.smithi046.lmxnwc (mgr.14164) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:05.345876+0000 mgr.smithi046.lmxnwc (mgr.14164) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:05.345876+0000 mgr.smithi046.lmxnwc (mgr.14164) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:07.346314+0000 mgr.smithi046.lmxnwc (mgr.14164) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:07.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:07.346314+0000 mgr.smithi046.lmxnwc (mgr.14164) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:09.346713+0000 mgr.smithi046.lmxnwc (mgr.14164) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:10.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:09.346713+0000 mgr.smithi046.lmxnwc (mgr.14164) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:11.347121+0000 mgr.smithi046.lmxnwc (mgr.14164) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:11.347121+0000 mgr.smithi046.lmxnwc (mgr.14164) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:13.347581+0000 mgr.smithi046.lmxnwc (mgr.14164) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:13.347581+0000 mgr.smithi046.lmxnwc (mgr.14164) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:15.348057+0000 mgr.smithi046.lmxnwc (mgr.14164) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:15.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:15.348057+0000 mgr.smithi046.lmxnwc (mgr.14164) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:17.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:17.348363+0000 mgr.smithi046.lmxnwc (mgr.14164) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:17.348363+0000 mgr.smithi046.lmxnwc (mgr.14164) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:19.348706+0000 mgr.smithi046.lmxnwc (mgr.14164) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:19.348706+0000 mgr.smithi046.lmxnwc (mgr.14164) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:13:20.749673+0000 mon.smithi046 (mon.0) 375 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:13:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:13:20.750766+0000 mon.smithi046 (mon.0) 376 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:13:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:13:20.749673+0000 mon.smithi046 (mon.0) 375 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:13:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:13:20.750766+0000 mon.smithi046 (mon.0) 376 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:13:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:21.349005+0000 mgr.smithi046.lmxnwc (mgr.14164) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:21.349005+0000 mgr.smithi046.lmxnwc (mgr.14164) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:23.349331+0000 mgr.smithi046.lmxnwc (mgr.14164) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:23.349331+0000 mgr.smithi046.lmxnwc (mgr.14164) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:25.349662+0000 mgr.smithi046.lmxnwc (mgr.14164) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:25.349662+0000 mgr.smithi046.lmxnwc (mgr.14164) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:27.350103+0000 mgr.smithi046.lmxnwc (mgr.14164) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:27.350103+0000 mgr.smithi046.lmxnwc (mgr.14164) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:29.350458+0000 mgr.smithi046.lmxnwc (mgr.14164) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:30.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:29.350458+0000 mgr.smithi046.lmxnwc (mgr.14164) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:31.350900+0000 mgr.smithi046.lmxnwc (mgr.14164) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:31.350900+0000 mgr.smithi046.lmxnwc (mgr.14164) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:33.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:33.351244+0000 mgr.smithi046.lmxnwc (mgr.14164) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:33.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:33.351244+0000 mgr.smithi046.lmxnwc (mgr.14164) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:35.351717+0000 mgr.smithi046.lmxnwc (mgr.14164) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:35.351717+0000 mgr.smithi046.lmxnwc (mgr.14164) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:37.352083+0000 mgr.smithi046.lmxnwc (mgr.14164) 274 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:37.352083+0000 mgr.smithi046.lmxnwc (mgr.14164) 274 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:39.352411+0000 mgr.smithi046.lmxnwc (mgr.14164) 275 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:40.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:39.352411+0000 mgr.smithi046.lmxnwc (mgr.14164) 275 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:41.352838+0000 mgr.smithi046.lmxnwc (mgr.14164) 276 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:41.352838+0000 mgr.smithi046.lmxnwc (mgr.14164) 276 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:43.353287+0000 mgr.smithi046.lmxnwc (mgr.14164) 277 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:43.353287+0000 mgr.smithi046.lmxnwc (mgr.14164) 277 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:45.353656+0000 mgr.smithi046.lmxnwc (mgr.14164) 278 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:45.353656+0000 mgr.smithi046.lmxnwc (mgr.14164) 278 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:47.354198+0000 mgr.smithi046.lmxnwc (mgr.14164) 279 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:47.354198+0000 mgr.smithi046.lmxnwc (mgr.14164) 279 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:49.354557+0000 mgr.smithi046.lmxnwc (mgr.14164) 280 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:49.354557+0000 mgr.smithi046.lmxnwc (mgr.14164) 280 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:51.354899+0000 mgr.smithi046.lmxnwc (mgr.14164) 281 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:51.354899+0000 mgr.smithi046.lmxnwc (mgr.14164) 281 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:53.355348+0000 mgr.smithi046.lmxnwc (mgr.14164) 282 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:53.355348+0000 mgr.smithi046.lmxnwc (mgr.14164) 282 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:55.355880+0000 mgr.smithi046.lmxnwc (mgr.14164) 283 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:55.355880+0000 mgr.smithi046.lmxnwc (mgr.14164) 283 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:13:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:57.356232+0000 mgr.smithi046.lmxnwc (mgr.14164) 284 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:13:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:13:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:57.356232+0000 mgr.smithi046.lmxnwc (mgr.14164) 284 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:13:59.356727+0000 mgr.smithi046.lmxnwc (mgr.14164) 285 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:13:59.356727+0000 mgr.smithi046.lmxnwc (mgr.14164) 285 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:01.357058+0000 mgr.smithi046.lmxnwc (mgr.14164) 286 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:01.357058+0000 mgr.smithi046.lmxnwc (mgr.14164) 286 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:03.357491+0000 mgr.smithi046.lmxnwc (mgr.14164) 287 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:03.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:03.357491+0000 mgr.smithi046.lmxnwc (mgr.14164) 287 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:05.357969+0000 mgr.smithi046.lmxnwc (mgr.14164) 288 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:05.357969+0000 mgr.smithi046.lmxnwc (mgr.14164) 288 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:07.358323+0000 mgr.smithi046.lmxnwc (mgr.14164) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:07.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:07.358323+0000 mgr.smithi046.lmxnwc (mgr.14164) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:09.358691+0000 mgr.smithi046.lmxnwc (mgr.14164) 290 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:10.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:09.358691+0000 mgr.smithi046.lmxnwc (mgr.14164) 290 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:11.359041+0000 mgr.smithi046.lmxnwc (mgr.14164) 291 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:11.359041+0000 mgr.smithi046.lmxnwc (mgr.14164) 291 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:13.359354+0000 mgr.smithi046.lmxnwc (mgr.14164) 292 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:13.359354+0000 mgr.smithi046.lmxnwc (mgr.14164) 292 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:15.359701+0000 mgr.smithi046.lmxnwc (mgr.14164) 293 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:15.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:15.359701+0000 mgr.smithi046.lmxnwc (mgr.14164) 293 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:17.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:17.360145+0000 mgr.smithi046.lmxnwc (mgr.14164) 294 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:17.360145+0000 mgr.smithi046.lmxnwc (mgr.14164) 294 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:19.360479+0000 mgr.smithi046.lmxnwc (mgr.14164) 295 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:19.360479+0000 mgr.smithi046.lmxnwc (mgr.14164) 295 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:14:20.751037+0000 mon.smithi046 (mon.0) 377 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:14:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:14:20.751982+0000 mon.smithi046 (mon.0) 378 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:14:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:14:20.751037+0000 mon.smithi046 (mon.0) 377 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:14:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:14:20.751982+0000 mon.smithi046 (mon.0) 378 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:14:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:21.360811+0000 mgr.smithi046.lmxnwc (mgr.14164) 296 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:21.360811+0000 mgr.smithi046.lmxnwc (mgr.14164) 296 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:23.361141+0000 mgr.smithi046.lmxnwc (mgr.14164) 297 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:23.361141+0000 mgr.smithi046.lmxnwc (mgr.14164) 297 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:25.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:25.361438+0000 mgr.smithi046.lmxnwc (mgr.14164) 298 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:25.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:25.361438+0000 mgr.smithi046.lmxnwc (mgr.14164) 298 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:27.361766+0000 mgr.smithi046.lmxnwc (mgr.14164) 299 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:27.361766+0000 mgr.smithi046.lmxnwc (mgr.14164) 299 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:29.362162+0000 mgr.smithi046.lmxnwc (mgr.14164) 300 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:30.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:29.362162+0000 mgr.smithi046.lmxnwc (mgr.14164) 300 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:31.362583+0000 mgr.smithi046.lmxnwc (mgr.14164) 301 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:31.362583+0000 mgr.smithi046.lmxnwc (mgr.14164) 301 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:32.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:14:32.904+0000 7fa239ffb700 1 -- 172.21.15.78:0/1220333115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2200031e0 con 0x7fa2440f6860 2024-01-21T08:14:32.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:14:32.904+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mon.1 v2:172.21.15.78:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7fa2200031e0 con 0x7fa2440f6860 2024-01-21T08:14:33.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:33.363035+0000 mgr.smithi046.lmxnwc (mgr.14164) 302 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:33.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:33.363035+0000 mgr.smithi046.lmxnwc (mgr.14164) 302 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:35.363452+0000 mgr.smithi046.lmxnwc (mgr.14164) 303 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:35.363452+0000 mgr.smithi046.lmxnwc (mgr.14164) 303 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:37.363858+0000 mgr.smithi046.lmxnwc (mgr.14164) 304 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:37.363858+0000 mgr.smithi046.lmxnwc (mgr.14164) 304 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:40.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:39.364264+0000 mgr.smithi046.lmxnwc (mgr.14164) 305 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:39.364264+0000 mgr.smithi046.lmxnwc (mgr.14164) 305 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:41.364667+0000 mgr.smithi046.lmxnwc (mgr.14164) 306 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:41.364667+0000 mgr.smithi046.lmxnwc (mgr.14164) 306 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:43.365090+0000 mgr.smithi046.lmxnwc (mgr.14164) 307 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:43.365090+0000 mgr.smithi046.lmxnwc (mgr.14164) 307 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:45.365506+0000 mgr.smithi046.lmxnwc (mgr.14164) 308 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:45.365506+0000 mgr.smithi046.lmxnwc (mgr.14164) 308 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:47.365952+0000 mgr.smithi046.lmxnwc (mgr.14164) 309 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:47.365952+0000 mgr.smithi046.lmxnwc (mgr.14164) 309 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:49.366296+0000 mgr.smithi046.lmxnwc (mgr.14164) 310 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:49.366296+0000 mgr.smithi046.lmxnwc (mgr.14164) 310 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:51.366721+0000 mgr.smithi046.lmxnwc (mgr.14164) 311 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:51.366721+0000 mgr.smithi046.lmxnwc (mgr.14164) 311 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:53.367041+0000 mgr.smithi046.lmxnwc (mgr.14164) 312 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:53.367041+0000 mgr.smithi046.lmxnwc (mgr.14164) 312 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:55.367457+0000 mgr.smithi046.lmxnwc (mgr.14164) 313 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:55.367457+0000 mgr.smithi046.lmxnwc (mgr.14164) 313 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:14:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:57.367849+0000 mgr.smithi046.lmxnwc (mgr.14164) 314 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:14:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:14:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:57.367849+0000 mgr.smithi046.lmxnwc (mgr.14164) 314 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:14:59.368194+0000 mgr.smithi046.lmxnwc (mgr.14164) 315 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:14:59.368194+0000 mgr.smithi046.lmxnwc (mgr.14164) 315 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:01.368648+0000 mgr.smithi046.lmxnwc (mgr.14164) 316 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:01.368648+0000 mgr.smithi046.lmxnwc (mgr.14164) 316 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:03.368944+0000 mgr.smithi046.lmxnwc (mgr.14164) 317 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:03.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:03.368944+0000 mgr.smithi046.lmxnwc (mgr.14164) 317 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:05.369323+0000 mgr.smithi046.lmxnwc (mgr.14164) 318 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:05.369323+0000 mgr.smithi046.lmxnwc (mgr.14164) 318 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:07.369821+0000 mgr.smithi046.lmxnwc (mgr.14164) 319 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:07.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:07.369821+0000 mgr.smithi046.lmxnwc (mgr.14164) 319 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:09.370169+0000 mgr.smithi046.lmxnwc (mgr.14164) 320 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:10.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:09.370169+0000 mgr.smithi046.lmxnwc (mgr.14164) 320 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:11.370762+0000 mgr.smithi046.lmxnwc (mgr.14164) 321 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:11.370762+0000 mgr.smithi046.lmxnwc (mgr.14164) 321 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:13.371094+0000 mgr.smithi046.lmxnwc (mgr.14164) 322 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:13.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:13.371094+0000 mgr.smithi046.lmxnwc (mgr.14164) 322 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:15.371583+0000 mgr.smithi046.lmxnwc (mgr.14164) 323 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:15.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:15.371583+0000 mgr.smithi046.lmxnwc (mgr.14164) 323 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:17.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:17.372141+0000 mgr.smithi046.lmxnwc (mgr.14164) 324 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:17.372141+0000 mgr.smithi046.lmxnwc (mgr.14164) 324 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:15:17.684701+0000 mgr.smithi046.lmxnwc (mgr.14164) 325 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:13:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:14:05Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:14:40Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:15:18.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('uid/gid not found') 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: uid/gid not found 2024-01-21T08:15:18.492 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: yield (conn, connr) 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: code, '\n'.join(err))) 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:13:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:14:05Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:14:40Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:15:18.493 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('uid/gid not found') 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: uid/gid not found 2024-01-21T08:15:18.494 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:17.685279+0000 mon.smithi046 (mon.0) 379 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:15:18.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:15:17.684701+0000 mgr.smithi046.lmxnwc (mgr.14164) 325 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:15:18.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:13:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:14:05Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:14:40Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:15:18.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('uid/gid not found') 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: uid/gid not found 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: yield (conn, connr) 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: code, '\n'.join(err))) 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:15:18.497 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:15:18.498 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:15:18.498 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:13:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.498 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:14:05Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.498 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:14:40Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:15:18.498 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:15:18.498 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:15:18.498 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:15:18.499 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:15:18.499 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:15:18.499 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:15:18.499 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:15:18.499 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('uid/gid not found') 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: uid/gid not found 2024-01-21T08:15:18.500 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:17.685279+0000 mon.smithi046 (mon.0) 379 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:15:19.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:18.680779+0000 mon.smithi046 (mon.0) 380 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-21T08:15:19.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:18.681430+0000 mon.smithi046 (mon.0) 381 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:15:19.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:15:18.681932+0000 mgr.smithi046.lmxnwc (mgr.14164) 326 : cephadm [INF] Deploying daemon osd.3 on smithi078 2024-01-21T08:15:19.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:18.836984+0000 mon.smithi046 (mon.0) 382 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:19.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:19.372472+0000 mgr.smithi046.lmxnwc (mgr.14164) 327 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:19.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:19.446910+0000 mon.smithi046 (mon.0) 383 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:18.680779+0000 mon.smithi046 (mon.0) 380 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-21T08:15:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:18.681430+0000 mon.smithi046 (mon.0) 381 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:15:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:15:18.681932+0000 mgr.smithi046.lmxnwc (mgr.14164) 326 : cephadm [INF] Deploying daemon osd.3 on smithi078 2024-01-21T08:15:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:18.836984+0000 mon.smithi046 (mon.0) 382 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:19.372472+0000 mgr.smithi046.lmxnwc (mgr.14164) 327 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:19.446910+0000 mon.smithi046 (mon.0) 383 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:21.074 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2024-01-21T08:15:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.071+0000 7fa23affd700 1 -- 172.21.15.78:0/1220333115 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa2440fdee0 con 0x7fa23005c2b0 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa23005c2b0 msgr2=0x7fa23005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa23005c2b0 0x7fa23005e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fa244105c40 tx=0x7fa23400b040).stop 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 msgr2=0x7fa2441063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2441063c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa240009f50 tx=0x7fa240009b30).stop 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 shutdown_connections 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa23005c2b0 0x7fa23005e770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2440f55c0 0x7fa244105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:15:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 --2- 172.21.15.78:0/1220333115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2440f6860 0x7fa2441063c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:15:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 >> 172.21.15.78:0/1220333115 conn(0x7fa2440f0d40 msgr2=0x7fa2440fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:15:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 shutdown_connections 2024-01-21T08:15:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:21.073+0000 7fa24c533700 1 -- 172.21.15.78:0/1220333115 wait complete. 2024-01-21T08:15:21.454 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.3.service 2024-01-21T08:15:21.456 INFO:tasks.cephadm:Deploying osd.4 on smithi078 with /dev/vg_nvme/lv_3... 2024-01-21T08:15:21.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-21T08:15:21.485 INFO:journalctl@ceph.osd.3.smithi078.stdout:-- Logs begin at Sun 2024-01-21 07:53:33 UTC. -- 2024-01-21T08:15:21.807 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:15:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3[60970]: debug 2024-01-21T08:15:21.544+0000 7fb797a96080 -1 Falling back to public interface 2024-01-21T08:15:21.807 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:20.752437+0000 mon.smithi046 (mon.0) 384 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:15:21.807 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:20.753174+0000 mon.smithi046 (mon.0) 385 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:15:21.807 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:21.067394+0000 mon.smithi046 (mon.0) 386 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:21.807 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:21.071359+0000 mon.smithi046 (mon.0) 387 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:21.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:20.752437+0000 mon.smithi046 (mon.0) 384 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:15:21.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:20.753174+0000 mon.smithi046 (mon.0) 385 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:15:21.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:21.067394+0000 mon.smithi046 (mon.0) 386 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:21.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:21.071359+0000 mon.smithi046 (mon.0) 387 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:22.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:21.073258+0000 mon.smithi046 (mon.0) 388 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-01-21T08:15:22.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:21.372753+0000 mgr.smithi046.lmxnwc (mgr.14164) 328 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:22.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:15:21.799117+0000 mgr.smithi046.lmxnwc (mgr.14164) 329 : cephadm [INF] Detected new or changed devices on smithi046 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:21.803529+0000 mon.smithi046 (mon.0) 389 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:22.055634+0000 mgr.smithi046.lmxnwc (mgr.14164) 330 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:22.058783+0000 mon.smithi046 (mon.0) 390 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:15:22.064531+0000 mgr.smithi046.lmxnwc (mgr.14164) 331 : cephadm [INF] Deploying daemon prometheus.smithi078 on smithi078 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:22.069718+0000 mon.smithi046 (mon.0) 391 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:22.077608+0000 mon.smithi046 (mon.0) 392 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:22.077678+0000 mon.smithi046 (mon.0) 393 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:22.077801+0000 mon.smithi046 (mon.0) 394 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:22.991 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:22.078586+0000 mon.smithi046 (mon.0) 395 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-01-21T08:15:22.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:21.073258+0000 mon.smithi046 (mon.0) 388 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-01-21T08:15:22.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:21.372753+0000 mgr.smithi046.lmxnwc (mgr.14164) 328 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:22.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:15:21.799117+0000 mgr.smithi046.lmxnwc (mgr.14164) 329 : cephadm [INF] Detected new or changed devices on smithi046 2024-01-21T08:15:22.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:21.803529+0000 mon.smithi046 (mon.0) 389 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:22.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:22.055634+0000 mgr.smithi046.lmxnwc (mgr.14164) 330 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:22.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:22.058783+0000 mon.smithi046 (mon.0) 390 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:15:22.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:15:22.064531+0000 mgr.smithi046.lmxnwc (mgr.14164) 331 : cephadm [INF] Deploying daemon prometheus.smithi078 on smithi078 2024-01-21T08:15:22.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:22.069718+0000 mon.smithi046 (mon.0) 391 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-01-21T08:15:22.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:22.077608+0000 mon.smithi046 (mon.0) 392 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-01-21T08:15:22.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:22.077678+0000 mon.smithi046 (mon.0) 393 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2024-01-21T08:15:22.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:22.077801+0000 mon.smithi046 (mon.0) 394 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:22.993 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:22.078586+0000 mon.smithi046 (mon.0) 395 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-01-21T08:15:23.014 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-21T08:15:23.014 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-21T08:15:23.015 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-01-21T08:15:23.015 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-01-21T08:15:23.015 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122572 s, 855 MB/s 2024-01-21T08:15:23.015 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-01-21T08:15:23.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2024-01-21T08:15:24.375 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:23.080838+0000 mon.smithi046 (mon.0) 396 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-01-21T08:15:24.375 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:23.080932+0000 mon.smithi046 (mon.0) 397 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2024-01-21T08:15:24.375 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:23.081126+0000 mon.smithi046 (mon.0) 398 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:24.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:23.080838+0000 mon.smithi046 (mon.0) 396 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-01-21T08:15:24.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:23.080932+0000 mon.smithi046 (mon.0) 397 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2024-01-21T08:15:24.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:23.081126+0000 mon.smithi046 (mon.0) 398 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:24.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 -- 172.21.15.78:0/907004866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 msgr2=0x7ff7a80fb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 --2- 172.21.15.78:0/907004866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a80fb700 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff798004660 tx=0x7ff798009b30).stop 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 -- 172.21.15.78:0/907004866 shutdown_connections 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 --2- 172.21.15.78:0/907004866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a80fbbf0 0x7ff7a80fdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 --2- 172.21.15.78:0/907004866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a80fb700 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 -- 172.21.15.78:0/907004866 >> 172.21.15.78:0/907004866 conn(0x7ff7a80f0d40 msgr2=0x7ff7a80f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 -- 172.21.15.78:0/907004866 shutdown_connections 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 -- 172.21.15.78:0/907004866 wait complete. 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.538+0000 7ff7af76e700 1 Processor -- start 2024-01-21T08:15:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7af76e700 1 -- start start 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7af76e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7af76e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a80fbbf0 0x7ff7a81096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7af76e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8109ce0 con 0x7ff7a80fbbf0 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7af76e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a810a590 con 0x7ff7a80602b0 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7ad50a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7ad50a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35284/0 (socket says 172.21.15.78:35284) 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7ad50a700 1 -- 172.21.15.78:0/2541597908 learned_addr learned my addr 172.21.15.78:0/2541597908 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7acd09700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a80fbbf0 0x7ff7a81096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7ad50a700 1 -- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a80fbbf0 msgr2=0x7ff7a81096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7ad50a700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a80fbbf0 0x7ff7a81096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7ad50a700 1 -- 172.21.15.78:0/2541597908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff798005040 con 0x7ff7a80602b0 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7ad50a700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a81091b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff798009830 tx=0x7ff798009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff79dffb700 1 -- 172.21.15.78:0/2541597908 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff798018070 con 0x7ff7a80602b0 2024-01-21T08:15:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff79dffb700 1 -- 172.21.15.78:0/2541597908 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff798013410 con 0x7ff7a80602b0 2024-01-21T08:15:24.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a810a810 con 0x7ff7a80602b0 2024-01-21T08:15:24.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.539+0000 7ff79dffb700 1 -- 172.21.15.78:0/2541597908 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79801c620 con 0x7ff7a80602b0 2024-01-21T08:15:24.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.540+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a810ad70 con 0x7ff7a80602b0 2024-01-21T08:15:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.541+0000 7ff79dffb700 1 -- 172.21.15.78:0/2541597908 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff79801c890 con 0x7ff7a80602b0 2024-01-21T08:15:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.541+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a80f49b0 con 0x7ff7a80602b0 2024-01-21T08:15:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.541+0000 7ff79dffb700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff79405c2b0 0x7ff79405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:15:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.541+0000 7ff79dffb700 1 -- 172.21.15.78:0/2541597908 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3261+0+0 (secure 0 0 0) 0x7ff798077560 con 0x7ff7a80602b0 2024-01-21T08:15:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.541+0000 7ff7acd09700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff79405c2b0 0x7ff79405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:15:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.542+0000 7ff7acd09700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff79405c2b0 0x7ff79405e770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7ff7a8060080 tx=0x7ff7a4008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:15:24.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.545+0000 7ff79dffb700 1 -- 172.21.15.78:0/2541597908 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff79804a9d0 con 0x7ff7a80602b0 2024-01-21T08:15:24.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:15:24.719+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7a81010f0 con 0x7ff79405c2b0 2024-01-21T08:15:25.341 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3[60970]: debug 2024-01-21T08:15:25.056+0000 7fb797a96080 -1 osd.3 0 log_to_monitors {default=true} 2024-01-21T08:15:25.341 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:24.056063+0000 mgr.smithi046.lmxnwc (mgr.14164) 332 : cluster [DBG] pgmap v313: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:25.341 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:24.085924+0000 mon.smithi046 (mon.0) 399 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-01-21T08:15:25.341 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:24.086097+0000 mon.smithi046 (mon.0) 400 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:25.341 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:24.722463+0000 mon.smithi046 (mon.0) 401 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:15:25.341 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:24.724355+0000 mon.smithi046 (mon.0) 402 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:15:25.341 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:24.725213+0000 mon.smithi046 (mon.0) 403 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:15:25.342 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:25.059891+0000 mon.smithi078 (mon.1) 6 : audit [INF] from='osd.3 [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-21T08:15:25.342 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:25.060359+0000 mon.smithi046 (mon.0) 404 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-21T08:15:25.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:24.056063+0000 mgr.smithi046.lmxnwc (mgr.14164) 332 : cluster [DBG] pgmap v313: 1 pgs: 1 creating+peering; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:25.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:24.085924+0000 mon.smithi046 (mon.0) 399 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-01-21T08:15:25.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:24.086097+0000 mon.smithi046 (mon.0) 400 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:25.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:24.722463+0000 mon.smithi046 (mon.0) 401 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:15:25.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:24.724355+0000 mon.smithi046 (mon.0) 402 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:15:25.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:24.725213+0000 mon.smithi046 (mon.0) 403 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:15:25.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:25.059891+0000 mon.smithi078 (mon.1) 6 : audit [INF] from='osd.3 [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-21T08:15:25.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:25.060359+0000 mon.smithi046 (mon.0) 404 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-21T08:15:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:24.721583+0000 mgr.smithi046.lmxnwc (mgr.14164) 333 : audit [DBG] from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:15:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:15:24.724055+0000 mgr.smithi046.lmxnwc (mgr.14164) 334 : cephadm [INF] Applying service osd.None on host smithi078... 2024-01-21T08:15:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:25.361863+0000 mon.smithi046 (mon.0) 405 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-21T08:15:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:25.361956+0000 mon.smithi046 (mon.0) 406 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-01-21T08:15:26.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:25.362105+0000 mon.smithi046 (mon.0) 407 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:26.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:25.363737+0000 mon.smithi078 (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:15:26.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:25.364306+0000 mon.smithi046 (mon.0) 408 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:15:26.741 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3[60970]: debug 2024-01-21T08:15:26.370+0000 7fb78e007700 -1 osd.3 0 waiting for initial osdmap 2024-01-21T08:15:26.741 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3[60970]: debug 2024-01-21T08:15:26.377+0000 7fb78877f700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-21T08:15:26.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:24.721583+0000 mgr.smithi046.lmxnwc (mgr.14164) 333 : audit [DBG] from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:15:26.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:15:24.724055+0000 mgr.smithi046.lmxnwc (mgr.14164) 334 : cephadm [INF] Applying service osd.None on host smithi078... 2024-01-21T08:15:26.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:25.361863+0000 mon.smithi046 (mon.0) 405 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-21T08:15:26.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:25.361956+0000 mon.smithi046 (mon.0) 406 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-01-21T08:15:26.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:25.362105+0000 mon.smithi046 (mon.0) 407 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:26.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:25.363737+0000 mon.smithi078 (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:15:26.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:25.364306+0000 mon.smithi046 (mon.0) 408 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:15:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:26.056430+0000 mgr.smithi046.lmxnwc (mgr.14164) 335 : cluster [DBG] pgmap v316: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:26.365197+0000 mon.smithi046 (mon.0) 409 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-01-21T08:15:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:26.365272+0000 mon.smithi046 (mon.0) 410 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-01-21T08:15:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:26.365829+0000 mon.smithi046 (mon.0) 411 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:26.369747+0000 mon.smithi046 (mon.0) 412 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:26.056430+0000 mgr.smithi046.lmxnwc (mgr.14164) 335 : cluster [DBG] pgmap v316: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-01-21T08:15:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:26.365197+0000 mon.smithi046 (mon.0) 409 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-01-21T08:15:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:26.365272+0000 mon.smithi046 (mon.0) 410 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-01-21T08:15:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:26.365829+0000 mon.smithi046 (mon.0) 411 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:27.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:26.369747+0000 mon.smithi046 (mon.0) 412 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:26.093600+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:15:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:26.093702+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:15:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:27.366461+0000 mon.smithi046 (mon.0) 413 : cluster [INF] osd.3 [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] boot 2024-01-21T08:15:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:27.366556+0000 mon.smithi046 (mon.0) 414 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-01-21T08:15:28.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:15:27.366719+0000 mon.smithi046 (mon.0) 415 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:28.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:28.368042+0000 mon.smithi046 (mon.0) 416 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-01-21T08:15:28.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:26.093600+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:15:28.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:26.093702+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:15:28.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:27.366461+0000 mon.smithi046 (mon.0) 413 : cluster [INF] osd.3 [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] boot 2024-01-21T08:15:28.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:27.366556+0000 mon.smithi046 (mon.0) 414 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-01-21T08:15:28.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:15:27.366719+0000 mon.smithi046 (mon.0) 415 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-21T08:15:28.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:28.368042+0000 mon.smithi046 (mon.0) 416 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-01-21T08:15:29.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:28.056834+0000 mgr.smithi046.lmxnwc (mgr.14164) 336 : cluster [DBG] pgmap v319: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:29.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:29.368278+0000 mon.smithi046 (mon.0) 417 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-01-21T08:15:29.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:28.056834+0000 mgr.smithi046.lmxnwc (mgr.14164) 336 : cluster [DBG] pgmap v319: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:29.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:29.368278+0000 mon.smithi046 (mon.0) 417 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-01-21T08:15:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:30.057145+0000 mgr.smithi046.lmxnwc (mgr.14164) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:30.057145+0000 mgr.smithi046.lmxnwc (mgr.14164) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:33.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:32.057598+0000 mgr.smithi046.lmxnwc (mgr.14164) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:33.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:32.057598+0000 mgr.smithi046.lmxnwc (mgr.14164) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:34.058127+0000 mgr.smithi046.lmxnwc (mgr.14164) 339 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:34.058127+0000 mgr.smithi046.lmxnwc (mgr.14164) 339 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:36.058663+0000 mgr.smithi046.lmxnwc (mgr.14164) 340 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:36.058663+0000 mgr.smithi046.lmxnwc (mgr.14164) 340 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:39.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:38.059189+0000 mgr.smithi046.lmxnwc (mgr.14164) 341 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:39.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:38.059189+0000 mgr.smithi046.lmxnwc (mgr.14164) 341 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:40.059720+0000 mgr.smithi046.lmxnwc (mgr.14164) 342 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:40.059720+0000 mgr.smithi046.lmxnwc (mgr.14164) 342 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:42.060026+0000 mgr.smithi046.lmxnwc (mgr.14164) 343 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:42.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:42.060026+0000 mgr.smithi046.lmxnwc (mgr.14164) 343 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:44.060379+0000 mgr.smithi046.lmxnwc (mgr.14164) 344 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:44.060379+0000 mgr.smithi046.lmxnwc (mgr.14164) 344 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:46.060761+0000 mgr.smithi046.lmxnwc (mgr.14164) 345 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:46.060761+0000 mgr.smithi046.lmxnwc (mgr.14164) 345 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:49.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:48.061151+0000 mgr.smithi046.lmxnwc (mgr.14164) 346 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:49.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:48.061151+0000 mgr.smithi046.lmxnwc (mgr.14164) 346 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:50.061659+0000 mgr.smithi046.lmxnwc (mgr.14164) 347 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:50.061659+0000 mgr.smithi046.lmxnwc (mgr.14164) 347 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:52.062059+0000 mgr.smithi046.lmxnwc (mgr.14164) 348 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:52.062059+0000 mgr.smithi046.lmxnwc (mgr.14164) 348 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:54.062392+0000 mgr.smithi046.lmxnwc (mgr.14164) 349 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:54.062392+0000 mgr.smithi046.lmxnwc (mgr.14164) 349 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:56.062762+0000 mgr.smithi046.lmxnwc (mgr.14164) 350 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:56.062762+0000 mgr.smithi046.lmxnwc (mgr.14164) 350 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:59.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:15:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:15:58.063190+0000 mgr.smithi046.lmxnwc (mgr.14164) 351 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:15:59.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:15:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:15:58.063190+0000 mgr.smithi046.lmxnwc (mgr.14164) 351 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:00.063571+0000 mgr.smithi046.lmxnwc (mgr.14164) 352 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:00.063571+0000 mgr.smithi046.lmxnwc (mgr.14164) 352 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:02.063973+0000 mgr.smithi046.lmxnwc (mgr.14164) 353 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:02.063973+0000 mgr.smithi046.lmxnwc (mgr.14164) 353 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:04.064436+0000 mgr.smithi046.lmxnwc (mgr.14164) 354 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:04.064436+0000 mgr.smithi046.lmxnwc (mgr.14164) 354 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:06.064794+0000 mgr.smithi046.lmxnwc (mgr.14164) 355 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:07.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:06.064794+0000 mgr.smithi046.lmxnwc (mgr.14164) 355 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:08.065283+0000 mgr.smithi046.lmxnwc (mgr.14164) 356 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:08.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:08.065283+0000 mgr.smithi046.lmxnwc (mgr.14164) 356 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:10.065713+0000 mgr.smithi046.lmxnwc (mgr.14164) 357 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:10.065713+0000 mgr.smithi046.lmxnwc (mgr.14164) 357 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:12.066104+0000 mgr.smithi046.lmxnwc (mgr.14164) 358 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:12.066104+0000 mgr.smithi046.lmxnwc (mgr.14164) 358 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:14.066673+0000 mgr.smithi046.lmxnwc (mgr.14164) 359 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:15.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:14.066673+0000 mgr.smithi046.lmxnwc (mgr.14164) 359 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:17.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:16.067042+0000 mgr.smithi046.lmxnwc (mgr.14164) 360 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:16.067042+0000 mgr.smithi046.lmxnwc (mgr.14164) 360 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:19.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:18.067479+0000 mgr.smithi046.lmxnwc (mgr.14164) 361 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:19.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:18.067479+0000 mgr.smithi046.lmxnwc (mgr.14164) 361 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:20.068004+0000 mgr.smithi046.lmxnwc (mgr.14164) 362 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:16:20.755062+0000 mon.smithi046 (mon.0) 418 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:16:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:16:20.755374+0000 mon.smithi046 (mon.0) 419 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:16:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:20.068004+0000 mgr.smithi046.lmxnwc (mgr.14164) 362 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:16:20.755062+0000 mon.smithi046 (mon.0) 418 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:16:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:16:20.755374+0000 mon.smithi046 (mon.0) 419 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:16:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:22.068375+0000 mgr.smithi046.lmxnwc (mgr.14164) 363 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:22.068375+0000 mgr.smithi046.lmxnwc (mgr.14164) 363 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:24.068899+0000 mgr.smithi046.lmxnwc (mgr.14164) 364 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:24.068899+0000 mgr.smithi046.lmxnwc (mgr.14164) 364 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:26.069299+0000 mgr.smithi046.lmxnwc (mgr.14164) 365 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:26.069299+0000 mgr.smithi046.lmxnwc (mgr.14164) 365 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:29.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:28.069896+0000 mgr.smithi046.lmxnwc (mgr.14164) 366 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:29.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:28.069896+0000 mgr.smithi046.lmxnwc (mgr.14164) 366 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:30.070420+0000 mgr.smithi046.lmxnwc (mgr.14164) 367 : cluster [DBG] pgmap v352: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:30.070420+0000 mgr.smithi046.lmxnwc (mgr.14164) 367 : cluster [DBG] pgmap v352: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:32.070953+0000 mgr.smithi046.lmxnwc (mgr.14164) 368 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:32.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:32 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:32.070953+0000 mgr.smithi046.lmxnwc (mgr.14164) 368 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:34.071439+0000 mgr.smithi046.lmxnwc (mgr.14164) 369 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:34.071439+0000 mgr.smithi046.lmxnwc (mgr.14164) 369 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:36.071760+0000 mgr.smithi046.lmxnwc (mgr.14164) 370 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:36.071760+0000 mgr.smithi046.lmxnwc (mgr.14164) 370 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:39.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:38.072088+0000 mgr.smithi046.lmxnwc (mgr.14164) 371 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:39.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:38.072088+0000 mgr.smithi046.lmxnwc (mgr.14164) 371 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:40.072574+0000 mgr.smithi046.lmxnwc (mgr.14164) 372 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:40.072574+0000 mgr.smithi046.lmxnwc (mgr.14164) 372 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:42.072992+0000 mgr.smithi046.lmxnwc (mgr.14164) 373 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:42.072992+0000 mgr.smithi046.lmxnwc (mgr.14164) 373 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:44.073331+0000 mgr.smithi046.lmxnwc (mgr.14164) 374 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:44.073331+0000 mgr.smithi046.lmxnwc (mgr.14164) 374 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:46.073879+0000 mgr.smithi046.lmxnwc (mgr.14164) 375 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:46.073879+0000 mgr.smithi046.lmxnwc (mgr.14164) 375 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:49.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:48.074361+0000 mgr.smithi046.lmxnwc (mgr.14164) 376 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:49.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:48.074361+0000 mgr.smithi046.lmxnwc (mgr.14164) 376 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:50.074768+0000 mgr.smithi046.lmxnwc (mgr.14164) 377 : cluster [DBG] pgmap v362: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:50.074768+0000 mgr.smithi046.lmxnwc (mgr.14164) 377 : cluster [DBG] pgmap v362: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:52.075298+0000 mgr.smithi046.lmxnwc (mgr.14164) 378 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:52.075298+0000 mgr.smithi046.lmxnwc (mgr.14164) 378 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:54.075880+0000 mgr.smithi046.lmxnwc (mgr.14164) 379 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:54.075880+0000 mgr.smithi046.lmxnwc (mgr.14164) 379 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:56.076397+0000 mgr.smithi046.lmxnwc (mgr.14164) 380 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:56.076397+0000 mgr.smithi046.lmxnwc (mgr.14164) 380 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:16:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:16:58.076827+0000 mgr.smithi046.lmxnwc (mgr.14164) 381 : cluster [DBG] pgmap v366: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:16:58.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:16:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:16:58.076827+0000 mgr.smithi046.lmxnwc (mgr.14164) 381 : cluster [DBG] pgmap v366: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:00.077165+0000 mgr.smithi046.lmxnwc (mgr.14164) 382 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:00.077165+0000 mgr.smithi046.lmxnwc (mgr.14164) 382 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:02.077603+0000 mgr.smithi046.lmxnwc (mgr.14164) 383 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:03.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:02.077603+0000 mgr.smithi046.lmxnwc (mgr.14164) 383 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:04.078063+0000 mgr.smithi046.lmxnwc (mgr.14164) 384 : cluster [DBG] pgmap v369: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:04.078063+0000 mgr.smithi046.lmxnwc (mgr.14164) 384 : cluster [DBG] pgmap v369: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:06.078509+0000 mgr.smithi046.lmxnwc (mgr.14164) 385 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:07.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:06.078509+0000 mgr.smithi046.lmxnwc (mgr.14164) 385 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:09.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:08.078858+0000 mgr.smithi046.lmxnwc (mgr.14164) 386 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:09.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:08.078858+0000 mgr.smithi046.lmxnwc (mgr.14164) 386 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:10.079205+0000 mgr.smithi046.lmxnwc (mgr.14164) 387 : cluster [DBG] pgmap v372: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:10.079205+0000 mgr.smithi046.lmxnwc (mgr.14164) 387 : cluster [DBG] pgmap v372: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:12.079560+0000 mgr.smithi046.lmxnwc (mgr.14164) 388 : cluster [DBG] pgmap v373: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:13.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:12.079560+0000 mgr.smithi046.lmxnwc (mgr.14164) 388 : cluster [DBG] pgmap v373: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:14.080033+0000 mgr.smithi046.lmxnwc (mgr.14164) 389 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:15.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:14.080033+0000 mgr.smithi046.lmxnwc (mgr.14164) 389 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:17.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:16.080375+0000 mgr.smithi046.lmxnwc (mgr.14164) 390 : cluster [DBG] pgmap v375: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:16.080375+0000 mgr.smithi046.lmxnwc (mgr.14164) 390 : cluster [DBG] pgmap v375: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:19.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:18.080845+0000 mgr.smithi046.lmxnwc (mgr.14164) 391 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:19.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:18.080845+0000 mgr.smithi046.lmxnwc (mgr.14164) 391 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:20.081185+0000 mgr.smithi046.lmxnwc (mgr.14164) 392 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:20.756632+0000 mon.smithi046 (mon.0) 420 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:17:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:20.756833+0000 mon.smithi046 (mon.0) 421 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:17:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:20.081185+0000 mgr.smithi046.lmxnwc (mgr.14164) 392 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:20.756632+0000 mon.smithi046 (mon.0) 420 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:17:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:20.756833+0000 mon.smithi046 (mon.0) 421 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:17:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:22.081682+0000 mgr.smithi046.lmxnwc (mgr.14164) 393 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:22.081682+0000 mgr.smithi046.lmxnwc (mgr.14164) 393 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:24.082210+0000 mgr.smithi046.lmxnwc (mgr.14164) 394 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:24.082210+0000 mgr.smithi046.lmxnwc (mgr.14164) 394 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:26.082562+0000 mgr.smithi046.lmxnwc (mgr.14164) 395 : cluster [DBG] pgmap v380: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:26.082562+0000 mgr.smithi046.lmxnwc (mgr.14164) 395 : cluster [DBG] pgmap v380: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:29.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:28.082953+0000 mgr.smithi046.lmxnwc (mgr.14164) 396 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:29.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:28.082953+0000 mgr.smithi046.lmxnwc (mgr.14164) 396 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:30.083427+0000 mgr.smithi046.lmxnwc (mgr.14164) 397 : cluster [DBG] pgmap v382: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:30.083427+0000 mgr.smithi046.lmxnwc (mgr.14164) 397 : cluster [DBG] pgmap v382: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:33.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:32.083928+0000 mgr.smithi046.lmxnwc (mgr.14164) 398 : cluster [DBG] pgmap v383: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:33.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:32.083928+0000 mgr.smithi046.lmxnwc (mgr.14164) 398 : cluster [DBG] pgmap v383: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:34.084291+0000 mgr.smithi046.lmxnwc (mgr.14164) 399 : cluster [DBG] pgmap v384: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:34.084291+0000 mgr.smithi046.lmxnwc (mgr.14164) 399 : cluster [DBG] pgmap v384: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:36.084797+0000 mgr.smithi046.lmxnwc (mgr.14164) 400 : cluster [DBG] pgmap v385: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:36.084797+0000 mgr.smithi046.lmxnwc (mgr.14164) 400 : cluster [DBG] pgmap v385: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:39.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:38.085247+0000 mgr.smithi046.lmxnwc (mgr.14164) 401 : cluster [DBG] pgmap v386: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:39.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:38.085247+0000 mgr.smithi046.lmxnwc (mgr.14164) 401 : cluster [DBG] pgmap v386: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:40.085754+0000 mgr.smithi046.lmxnwc (mgr.14164) 402 : cluster [DBG] pgmap v387: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:40.085754+0000 mgr.smithi046.lmxnwc (mgr.14164) 402 : cluster [DBG] pgmap v387: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:43.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:42.086249+0000 mgr.smithi046.lmxnwc (mgr.14164) 403 : cluster [DBG] pgmap v388: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:42.086249+0000 mgr.smithi046.lmxnwc (mgr.14164) 403 : cluster [DBG] pgmap v388: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:45.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:44.086701+0000 mgr.smithi046.lmxnwc (mgr.14164) 404 : cluster [DBG] pgmap v389: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:44.086701+0000 mgr.smithi046.lmxnwc (mgr.14164) 404 : cluster [DBG] pgmap v389: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:46.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:46.087102+0000 mgr.smithi046.lmxnwc (mgr.14164) 405 : cluster [DBG] pgmap v390: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:46.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:46 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:46.087102+0000 mgr.smithi046.lmxnwc (mgr.14164) 405 : cluster [DBG] pgmap v390: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:49.447 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:48.087569+0000 mgr.smithi046.lmxnwc (mgr.14164) 406 : cluster [DBG] pgmap v391: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:49.447 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:48.606475+0000 mon.smithi046 (mon.0) 422 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-21T08:17:49.447 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:48.607366+0000 mon.smithi046 (mon.0) 423 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-21T08:17:49.447 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:48.608135+0000 mon.smithi046 (mon.0) 424 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:49.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:48.087569+0000 mgr.smithi046.lmxnwc (mgr.14164) 406 : cluster [DBG] pgmap v391: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:49.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:48.606475+0000 mon.smithi046 (mon.0) 422 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-21T08:17:49.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:48.607366+0000 mon.smithi046 (mon.0) 423 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-21T08:17:49.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:48.608135+0000 mon.smithi046 (mon.0) 424 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:48.605762+0000 mgr.smithi046.lmxnwc (mgr.14164) 407 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:15:58Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:16:36Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:17:13Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:17:50.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:17:50.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('uid/gid not found') 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: uid/gid not found 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: yield (conn, connr) 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: code, '\n'.join(err))) 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:15:58Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:16:36Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr time="2024-01-21T08:17:13Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:17:50.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:17:50.745 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:17:50.745 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:17:50.745 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:17:50.745 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:17:50.745 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('uid/gid not found') 2024-01-21T08:17:50.745 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: uid/gid not found 2024-01-21T08:17:50.745 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:48.606138+0000 mgr.smithi046.lmxnwc (mgr.14164) 408 : cephadm [INF] Reconfiguring mon.smithi046 (unknown last config time)... 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:48.608782+0000 mgr.smithi046.lmxnwc (mgr.14164) 409 : cephadm [INF] Reconfiguring daemon mon.smithi046 on smithi046 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.493390+0000 mon.smithi078 (mon.1) 8 : audit [INF] from='client.? 172.21.15.78:0/377205376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a04900e5-161b-4cb2-be30-fd8513a1c2b2"}]: dispatch 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.493915+0000 mon.smithi046 (mon.0) 425 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a04900e5-161b-4cb2-be30-fd8513a1c2b2"}]: dispatch 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.497744+0000 mon.smithi046 (mon.0) 426 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a04900e5-161b-4cb2-be30-fd8513a1c2b2"}]': finished 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:49.497795+0000 mon.smithi046 (mon.0) 427 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.497919+0000 mon.smithi046 (mon.0) 428 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.674203+0000 mon.smithi046 (mon.0) 429 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.674911+0000 mon.smithi046 (mon.0) 430 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi046.lmxnwc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.675546+0000 mon.smithi046 (mon.0) 431 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.676137+0000 mon.smithi046 (mon.0) 432 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:50.746 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:49.905158+0000 mon.smithi078 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.78:0/1344088122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:48.605762+0000 mgr.smithi046.lmxnwc (mgr.14164) 407 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:15:58Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:16:36Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:17:13Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:17:50.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('uid/gid not found') 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: uid/gid not found 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: yield (conn, connr) 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: code, '\n'.join(err))) 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi078 ... 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-21T08:17:50.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:15:58Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:16:36Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr time="2024-01-21T08:17:13Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:17:50.749 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('uid/gid not found') 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: uid/gid not found 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:48.606138+0000 mgr.smithi046.lmxnwc (mgr.14164) 408 : cephadm [INF] Reconfiguring mon.smithi046 (unknown last config time)... 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:48.608782+0000 mgr.smithi046.lmxnwc (mgr.14164) 409 : cephadm [INF] Reconfiguring daemon mon.smithi046 on smithi046 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.493390+0000 mon.smithi078 (mon.1) 8 : audit [INF] from='client.? 172.21.15.78:0/377205376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a04900e5-161b-4cb2-be30-fd8513a1c2b2"}]: dispatch 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.493915+0000 mon.smithi046 (mon.0) 425 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a04900e5-161b-4cb2-be30-fd8513a1c2b2"}]: dispatch 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.497744+0000 mon.smithi046 (mon.0) 426 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a04900e5-161b-4cb2-be30-fd8513a1c2b2"}]': finished 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:49.497795+0000 mon.smithi046 (mon.0) 427 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-01-21T08:17:50.750 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.497919+0000 mon.smithi046 (mon.0) 428 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:17:50.751 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.674203+0000 mon.smithi046 (mon.0) 429 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:50.751 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.674911+0000 mon.smithi046 (mon.0) 430 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi046.lmxnwc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-21T08:17:50.751 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.675546+0000 mon.smithi046 (mon.0) 431 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-21T08:17:50.751 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.676137+0000 mon.smithi046 (mon.0) 432 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:50.751 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:49.905158+0000 mon.smithi078 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.78:0/1344088122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:17:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:49.674704+0000 mgr.smithi046.lmxnwc (mgr.14164) 410 : cephadm [INF] Reconfiguring mgr.smithi046.lmxnwc (unknown last config time)... 2024-01-21T08:17:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:49.676658+0000 mgr.smithi046.lmxnwc (mgr.14164) 411 : cephadm [INF] Reconfiguring daemon mgr.smithi046.lmxnwc on smithi046 2024-01-21T08:17:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:50.087933+0000 mgr.smithi046.lmxnwc (mgr.14164) 412 : cluster [DBG] pgmap v393: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:50.648507+0000 mon.smithi046 (mon.0) 433 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:49.674704+0000 mgr.smithi046.lmxnwc (mgr.14164) 410 : cephadm [INF] Reconfiguring mgr.smithi046.lmxnwc (unknown last config time)... 2024-01-21T08:17:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:49.676658+0000 mgr.smithi046.lmxnwc (mgr.14164) 411 : cephadm [INF] Reconfiguring daemon mgr.smithi046.lmxnwc on smithi046 2024-01-21T08:17:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:50.087933+0000 mgr.smithi046.lmxnwc (mgr.14164) 412 : cluster [DBG] pgmap v393: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:50.648507+0000 mon.smithi046 (mon.0) 433 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:52.729 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:50.649114+0000 mgr.smithi046.lmxnwc (mgr.14164) 413 : cephadm [INF] Reconfiguring alertmanager.smithi046 (dependencies changed)... 2024-01-21T08:17:52.729 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:50.650983+0000 mgr.smithi046.lmxnwc (mgr.14164) 414 : cephadm [INF] Reconfiguring daemon alertmanager.smithi046 on smithi046 2024-01-21T08:17:52.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:52 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:50.649114+0000 mgr.smithi046.lmxnwc (mgr.14164) 413 : cephadm [INF] Reconfiguring alertmanager.smithi046 (dependencies changed)... 2024-01-21T08:17:52.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:52 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:50.650983+0000 mgr.smithi046.lmxnwc (mgr.14164) 414 : cephadm [INF] Reconfiguring daemon alertmanager.smithi046 on smithi046 2024-01-21T08:17:53.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:52.088223+0000 mgr.smithi046.lmxnwc (mgr.14164) 415 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:53.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:53.331444+0000 mon.smithi046 (mon.0) 434 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:53.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:53.332186+0000 mon.smithi046 (mon.0) 435 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T08:17:53.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:53.332837+0000 mon.smithi046 (mon.0) 436 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:53.645 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:52.088223+0000 mgr.smithi046.lmxnwc (mgr.14164) 415 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:53.645 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:53.331444+0000 mon.smithi046 (mon.0) 434 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:53.645 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:53.332186+0000 mon.smithi046 (mon.0) 435 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi046", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T08:17:53.645 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:53.332837+0000 mon.smithi046 (mon.0) 436 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:54.470 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:53.331959+0000 mgr.smithi046.lmxnwc (mgr.14164) 416 : cephadm [INF] Reconfiguring crash.smithi046 (monmap changed)... 2024-01-21T08:17:54.470 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:53.333291+0000 mgr.smithi046.lmxnwc (mgr.14164) 417 : cephadm [INF] Reconfiguring daemon crash.smithi046 on smithi046 2024-01-21T08:17:54.470 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:54.198746+0000 mon.smithi046 (mon.0) 437 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:53.331959+0000 mgr.smithi046.lmxnwc (mgr.14164) 416 : cephadm [INF] Reconfiguring crash.smithi046 (monmap changed)... 2024-01-21T08:17:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:53.333291+0000 mgr.smithi046.lmxnwc (mgr.14164) 417 : cephadm [INF] Reconfiguring daemon crash.smithi046 on smithi046 2024-01-21T08:17:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:54.198746+0000 mon.smithi046 (mon.0) 437 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:55.661 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:54.088712+0000 mgr.smithi046.lmxnwc (mgr.14164) 418 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:55.661 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:54.199291+0000 mgr.smithi046.lmxnwc (mgr.14164) 419 : cephadm [INF] Reconfiguring node-exporter.smithi046 (unknown last config time)... 2024-01-21T08:17:55.662 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:54.199398+0000 mgr.smithi046.lmxnwc (mgr.14164) 420 : cephadm [INF] Reconfiguring daemon node-exporter.smithi046 on smithi046 2024-01-21T08:17:55.662 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:55.317690+0000 mon.smithi046 (mon.0) 438 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-21T08:17:55.662 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:55.318549+0000 mon.smithi046 (mon.0) 439 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:54.088712+0000 mgr.smithi046.lmxnwc (mgr.14164) 418 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:54.199291+0000 mgr.smithi046.lmxnwc (mgr.14164) 419 : cephadm [INF] Reconfiguring node-exporter.smithi046 (unknown last config time)... 2024-01-21T08:17:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:54.199398+0000 mgr.smithi046.lmxnwc (mgr.14164) 420 : cephadm [INF] Reconfiguring daemon node-exporter.smithi046 on smithi046 2024-01-21T08:17:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:55.317690+0000 mon.smithi046 (mon.0) 438 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-21T08:17:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:55.318549+0000 mon.smithi046 (mon.0) 439 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:17:56.721 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:17:55.319223+0000 mgr.smithi046.lmxnwc (mgr.14164) 421 : cephadm [INF] Deploying daemon osd.4 on smithi078 2024-01-21T08:17:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:17:55.319223+0000 mgr.smithi046.lmxnwc (mgr.14164) 421 : cephadm [INF] Deploying daemon osd.4 on smithi078 2024-01-21T08:17:57.695 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:56.089013+0000 mgr.smithi046.lmxnwc (mgr.14164) 422 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:56.089013+0000 mgr.smithi046.lmxnwc (mgr.14164) 422 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:57.769 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 4 on host 'smithi078' 2024-01-21T08:17:57.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.766+0000 7ff79dffb700 1 -- 172.21.15.78:0/2541597908 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff7a81010f0 con 0x7ff79405c2b0 2024-01-21T08:17:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff79405c2b0 msgr2=0x7ff79405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:17:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff79405c2b0 0x7ff79405e770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7ff7a8060080 tx=0x7ff7a4008040).stop 2024-01-21T08:17:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 msgr2=0x7ff7a81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:17:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a81091b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff798009830 tx=0x7ff798009b30).stop 2024-01-21T08:17:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 shutdown_connections 2024-01-21T08:17:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff79405c2b0 0x7ff79405e770 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:17:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7a80602b0 0x7ff7a81091b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:17:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 --2- 172.21.15.78:0/2541597908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a80fbbf0 0x7ff7a81096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:17:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 >> 172.21.15.78:0/2541597908 conn(0x7ff7a80f0d40 msgr2=0x7ff7a80f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:17:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 shutdown_connections 2024-01-21T08:17:57.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:17:57.768+0000 7ff7af76e700 1 -- 172.21.15.78:0/2541597908 wait complete. 2024-01-21T08:17:58.100 DEBUG:teuthology.orchestra.run.smithi078:osd.4> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.4.service 2024-01-21T08:17:58.102 INFO:tasks.cephadm:Deploying osd.5 on smithi078 with /dev/vg_nvme/lv_2... 2024-01-21T08:17:58.103 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-21T08:17:58.133 INFO:journalctl@ceph.osd.4.smithi078.stdout:-- Logs begin at Sun 2024-01-21 07:53:33 UTC. -- 2024-01-21T08:17:58.479 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:17:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4[64226]: debug 2024-01-21T08:17:58.222+0000 7f3bccc47080 -1 Falling back to public interface 2024-01-21T08:17:58.932 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:57.762451+0000 mon.smithi046 (mon.0) 440 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:58.932 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:17:57.766392+0000 mon.smithi046 (mon.0) 441 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:58.932 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:17:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:17:58.089500+0000 mgr.smithi046.lmxnwc (mgr.14164) 423 : cluster [DBG] pgmap v397: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:59.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:57.762451+0000 mon.smithi046 (mon.0) 440 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:59.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:17:57.766392+0000 mon.smithi046 (mon.0) 441 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:17:59.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:17:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:17:58.089500+0000 mgr.smithi046.lmxnwc (mgr.14164) 423 : cluster [DBG] pgmap v397: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:17:59.589 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-21T08:17:59.589 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-21T08:17:59.589 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-01-21T08:17:59.589 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-01-21T08:17:59.589 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0152636 s, 687 MB/s 2024-01-21T08:17:59.589 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-01-21T08:17:59.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2024-01-21T08:18:01.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/3816898535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480f9880 msgr2=0x7f9a480fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:01.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/3816898535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480f9880 0x7f9a480fbc70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9a38004660 tx=0x7f9a38009b30).stop 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/3816898535 shutdown_connections 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/3816898535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480f9880 0x7f9a480fbc70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/3816898535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a480602b0 0x7f9a480f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/3816898535 >> 172.21.15.78:0/3816898535 conn(0x7f9a480f0d40 msgr2=0x7f9a480f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/3816898535 shutdown_connections 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/3816898535 wait complete. 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.179+0000 7f9a4d4a4700 1 Processor -- start 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a4d4a4700 1 -- start start 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a4d4a4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480602b0 0x7f9a480f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:01.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a4d4a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a480f9880 0x7f9a480f5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a4d4a4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480f57f0 con 0x7f9a480f9880 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a4d4a4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480f5930 con 0x7f9a480602b0 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a46ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480602b0 0x7f9a480f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a46ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480602b0 0x7f9a480f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45870/0 (socket says 172.21.15.78:45870) 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a46ffd700 1 -- 172.21.15.78:0/75565776 learned_addr learned my addr 172.21.15.78:0/75565776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a46ffd700 1 -- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a480f9880 msgr2=0x7f9a480f5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a46ffd700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a480f9880 0x7f9a480f5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a46ffd700 1 -- 172.21.15.78:0/75565776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a38005040 con 0x7f9a480602b0 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a46ffd700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480602b0 0x7f9a480f6d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9a3c00b410 tx=0x7f9a3c006ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a2f7fe700 1 -- 172.21.15.78:0/75565776 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a3c0125f0 con 0x7f9a480602b0 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a480f5c10 con 0x7f9a480602b0 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a2f7fe700 1 -- 172.21.15.78:0/75565776 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a3c012c00 con 0x7f9a480602b0 2024-01-21T08:18:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a2f7fe700 1 -- 172.21.15.78:0/75565776 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a3c002aa0 con 0x7f9a480602b0 2024-01-21T08:18:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.180+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4810efb0 con 0x7f9a480602b0 2024-01-21T08:18:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.181+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a48045ad0 con 0x7f9a480602b0 2024-01-21T08:18:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.181+0000 7f9a2f7fe700 1 -- 172.21.15.78:0/75565776 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9a3c00f030 con 0x7f9a480602b0 2024-01-21T08:18:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.182+0000 7f9a2f7fe700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a3005c2b0 0x7f9a3005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.182+0000 7f9a2f7fe700 1 -- 172.21.15.78:0/75565776 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3803+0+0 (secure 0 0 0) 0x7f9a3c075d70 con 0x7f9a480602b0 2024-01-21T08:18:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.182+0000 7f9a467fc700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a3005c2b0 0x7f9a3005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.182+0000 7f9a467fc700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a3005c2b0 0x7f9a3005e770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9a38009f50 tx=0x7f9a38009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.186+0000 7f9a2f7fe700 1 -- 172.21.15.78:0/75565776 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a3c049000 con 0x7f9a480602b0 2024-01-21T08:18:01.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:01.361+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a48057f10 con 0x7f9a3005c2b0 2024-01-21T08:18:01.659 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:00.089991+0000 mgr.smithi046.lmxnwc (mgr.14164) 424 : cluster [DBG] pgmap v398: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:18:01.659 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:01.365594+0000 mon.smithi046 (mon.0) 442 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:18:01.660 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:01.367366+0000 mon.smithi046 (mon.0) 443 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:18:01.660 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:01.368096+0000 mon.smithi046 (mon.0) 444 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:18:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:00.089991+0000 mgr.smithi046.lmxnwc (mgr.14164) 424 : cluster [DBG] pgmap v398: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:18:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:01.365594+0000 mon.smithi046 (mon.0) 442 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-21T08:18:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:01.367366+0000 mon.smithi046 (mon.0) 443 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-21T08:18:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:01.368096+0000 mon.smithi046 (mon.0) 444 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:18:01.992 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:18:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4[64226]: debug 2024-01-21T08:18:01.726+0000 7f3bccc47080 -1 osd.4 0 log_to_monitors {default=true} 2024-01-21T08:18:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:01.364727+0000 mgr.smithi046.lmxnwc (mgr.14164) 425 : audit [DBG] from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:18:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:18:01.367087+0000 mgr.smithi046.lmxnwc (mgr.14164) 426 : cephadm [INF] Applying service osd.None on host smithi078... 2024-01-21T08:18:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:01.731471+0000 mon.smithi078 (mon.1) 10 : audit [INF] from='osd.4 [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-21T08:18:02.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:01.731924+0000 mon.smithi046 (mon.0) 445 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-21T08:18:02.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:01.364727+0000 mgr.smithi046.lmxnwc (mgr.14164) 425 : audit [DBG] from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T08:18:02.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:18:01.367087+0000 mgr.smithi046.lmxnwc (mgr.14164) 426 : cephadm [INF] Applying service osd.None on host smithi078... 2024-01-21T08:18:02.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:01.731471+0000 mon.smithi078 (mon.1) 10 : audit [INF] from='osd.4 [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-21T08:18:02.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:01.731924+0000 mon.smithi046 (mon.0) 445 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-21T08:18:03.241 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:18:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4[64226]: debug 2024-01-21T08:18:02.980+0000 7f3bc31b8700 -1 osd.4 0 waiting for initial osdmap 2024-01-21T08:18:03.242 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:18:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4[64226]: debug 2024-01-21T08:18:02.988+0000 7f3bbf934700 -1 osd.4 30 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-21T08:18:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:02.090490+0000 mgr.smithi046.lmxnwc (mgr.14164) 427 : cluster [DBG] pgmap v399: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:18:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.400828+0000 mon.smithi046 (mon.0) 446 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-21T08:18:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:02.400912+0000 mon.smithi046 (mon.0) 447 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-01-21T08:18:03.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.401039+0000 mon.smithi046 (mon.0) 448 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.405710+0000 mon.smithi078 (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.406468+0000 mon.smithi046 (mon.0) 449 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.973581+0000 mon.smithi078 (mon.1) 12 : audit [INF] from='client.? 172.21.15.78:0/740649067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f2f136-72a2-4e01-901e-e2e244f3cf89"}]: dispatch 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.974166+0000 mon.smithi046 (mon.0) 450 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f2f136-72a2-4e01-901e-e2e244f3cf89"}]: dispatch 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.979133+0000 mon.smithi046 (mon.0) 451 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.979225+0000 mon.smithi046 (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f2f136-72a2-4e01-901e-e2e244f3cf89"}]': finished 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:02.979278+0000 mon.smithi046 (mon.0) 453 : cluster [DBG] osdmap e30: 6 total, 4 up, 6 in 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.979413+0000 mon.smithi046 (mon.0) 454 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.979843+0000 mon.smithi046 (mon.0) 455 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:03.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:02.982296+0000 mon.smithi046 (mon.0) 456 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:02.090490+0000 mgr.smithi046.lmxnwc (mgr.14164) 427 : cluster [DBG] pgmap v399: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.400828+0000 mon.smithi046 (mon.0) 446 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:02.400912+0000 mon.smithi046 (mon.0) 447 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.401039+0000 mon.smithi046 (mon.0) 448 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.405710+0000 mon.smithi078 (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.406468+0000 mon.smithi046 (mon.0) 449 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.973581+0000 mon.smithi078 (mon.1) 12 : audit [INF] from='client.? 172.21.15.78:0/740649067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f2f136-72a2-4e01-901e-e2e244f3cf89"}]: dispatch 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.974166+0000 mon.smithi046 (mon.0) 450 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f2f136-72a2-4e01-901e-e2e244f3cf89"}]: dispatch 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.979133+0000 mon.smithi046 (mon.0) 451 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.979225+0000 mon.smithi046 (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f2f136-72a2-4e01-901e-e2e244f3cf89"}]': finished 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:02.979278+0000 mon.smithi046 (mon.0) 453 : cluster [DBG] osdmap e30: 6 total, 4 up, 6 in 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.979413+0000 mon.smithi046 (mon.0) 454 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:03.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.979843+0000 mon.smithi046 (mon.0) 455 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:03.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:02.982296+0000 mon.smithi046 (mon.0) 456 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:04.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:03.403746+0000 mon.smithi078 (mon.1) 13 : audit [DBG] from='client.? 172.21.15.78:0/2002924356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:18:04.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:03.980362+0000 mon.smithi046 (mon.0) 457 : cluster [INF] osd.4 [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] boot 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:03.980404+0000 mon.smithi046 (mon.0) 458 : cluster [DBG] osdmap e31: 6 total, 5 up, 6 in 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:03.980536+0000 mon.smithi046 (mon.0) 459 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:03.980897+0000 mon.smithi046 (mon.0) 460 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:03.403746+0000 mon.smithi078 (mon.1) 13 : audit [DBG] from='client.? 172.21.15.78:0/2002924356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:03.980362+0000 mon.smithi046 (mon.0) 457 : cluster [INF] osd.4 [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] boot 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:03.980404+0000 mon.smithi046 (mon.0) 458 : cluster [DBG] osdmap e31: 6 total, 5 up, 6 in 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:03.980536+0000 mon.smithi046 (mon.0) 459 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-21T08:18:04.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:03.980897+0000 mon.smithi046 (mon.0) 460 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:02.722149+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:18:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:02.722246+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:18:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:04.090900+0000 mgr.smithi046.lmxnwc (mgr.14164) 428 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:02.722149+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:18:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:02.722246+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:18:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:04.090900+0000 mgr.smithi046.lmxnwc (mgr.14164) 428 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:05.411109+0000 mon.smithi046 (mon.0) 461 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2024-01-21T08:18:06.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:05.411246+0000 mon.smithi046 (mon.0) 462 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:06.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:05.411109+0000 mon.smithi046 (mon.0) 461 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2024-01-21T08:18:06.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:05.411246+0000 mon.smithi046 (mon.0) 462 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:07.692 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:06.091358+0000 mgr.smithi046.lmxnwc (mgr.14164) 429 : cluster [DBG] pgmap v405: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:06.091358+0000 mgr.smithi046.lmxnwc (mgr.14164) 429 : cluster [DBG] pgmap v405: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:09.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:08.091856+0000 mgr.smithi046.lmxnwc (mgr.14164) 430 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:09.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:09.074200+0000 mon.smithi046 (mon.0) 463 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-21T08:18:09.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:09.075106+0000 mon.smithi046 (mon.0) 464 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:18:09.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:08.091856+0000 mgr.smithi046.lmxnwc (mgr.14164) 430 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:09.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:09.074200+0000 mon.smithi046 (mon.0) 463 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-21T08:18:09.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:09.075106+0000 mon.smithi046 (mon.0) 464 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:18:10.726 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:18:09.075834+0000 mgr.smithi046.lmxnwc (mgr.14164) 431 : cephadm [INF] Deploying daemon osd.5 on smithi078 2024-01-21T08:18:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:18:09.075834+0000 mgr.smithi046.lmxnwc (mgr.14164) 431 : cephadm [INF] Deploying daemon osd.5 on smithi078 2024-01-21T08:18:11.393 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 5 on host 'smithi078' 2024-01-21T08:18:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.389+0000 7f9a2f7fe700 1 -- 172.21.15.78:0/75565776 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9a48057f10 con 0x7f9a3005c2b0 2024-01-21T08:18:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a3005c2b0 msgr2=0x7f9a3005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a3005c2b0 0x7f9a3005e770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9a38009f50 tx=0x7f9a38009b30).stop 2024-01-21T08:18:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480602b0 msgr2=0x7f9a480f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480602b0 0x7f9a480f6d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9a3c00b410 tx=0x7f9a3c006ef0).stop 2024-01-21T08:18:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 shutdown_connections 2024-01-21T08:18:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a3005c2b0 0x7f9a3005e770 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:11.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a480602b0 0x7f9a480f6d80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:11.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 --2- 172.21.15.78:0/75565776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a480f9880 0x7f9a480f5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:11.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 >> 172.21.15.78:0/75565776 conn(0x7f9a480f0d40 msgr2=0x7f9a480f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:11.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 shutdown_connections 2024-01-21T08:18:11.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:11.392+0000 7f9a4d4a4700 1 -- 172.21.15.78:0/75565776 wait complete. 2024-01-21T08:18:11.685 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:10.092225+0000 mgr.smithi046.lmxnwc (mgr.14164) 432 : cluster [DBG] pgmap v407: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:11.686 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:11.385820+0000 mon.smithi046 (mon.0) 465 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:18:11.686 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:11.389522+0000 mon.smithi046 (mon.0) 466 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:18:11.719 DEBUG:teuthology.orchestra.run.smithi078:osd.5> sudo journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.5.service 2024-01-21T08:18:11.721 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-01-21T08:18:11.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd stat -f json 2024-01-21T08:18:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:10.092225+0000 mgr.smithi046.lmxnwc (mgr.14164) 432 : cluster [DBG] pgmap v407: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:11.385820+0000 mon.smithi046 (mon.0) 465 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:18:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:11.389522+0000 mon.smithi046 (mon.0) 466 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:18:11.751 INFO:journalctl@ceph.osd.5.smithi078.stdout:-- Logs begin at Sun 2024-01-21 07:53:33 UTC. -- 2024-01-21T08:18:11.991 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:18:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5[67357]: debug 2024-01-21T08:18:11.839+0000 7f55da08d080 -1 Falling back to public interface 2024-01-21T08:18:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:12.092512+0000 mgr.smithi046.lmxnwc (mgr.14164) 433 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:12.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:12 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:12.092512+0000 mgr.smithi046.lmxnwc (mgr.14164) 433 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1689032120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f55c0 msgr2=0x7fd5a80f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1689032120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f55c0 0x7fd5a80f59e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fd598004660 tx=0x7fd598009b30).stop 2024-01-21T08:18:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1689032120 shutdown_connections 2024-01-21T08:18:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1689032120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5a80f6860 0x7fd5a80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1689032120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f55c0 0x7fd5a80f59e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1689032120 >> 172.21.15.46:0/1689032120 conn(0x7fd5a80f0d40 msgr2=0x7fd5a80f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1689032120 shutdown_connections 2024-01-21T08:18:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1689032120 wait complete. 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.978+0000 7fd5ae9a3700 1 Processor -- start 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5ae9a3700 1 -- start start 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5ae9a3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5a80f55c0 0x7fd5a8109cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5ae9a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f6860 0x7fd5a810c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5ae9a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80603c0 con 0x7fd5a80f6860 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5ae9a3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8060500 con 0x7fd5a80f55c0 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f6860 0x7fd5a810c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f6860 0x7fd5a810c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40804/0 (socket says 172.21.15.46:40804) 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a77fe700 1 -- 172.21.15.46:0/1972896418 learned_addr learned my addr 172.21.15.46:0/1972896418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a7fff700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5a80f55c0 0x7fd5a8109cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a77fe700 1 -- 172.21.15.46:0/1972896418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5a80f55c0 msgr2=0x7fd5a8109cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a77fe700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5a80f55c0 0x7fd5a8109cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a77fe700 1 -- 172.21.15.46:0/1972896418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598005040 con 0x7fd5a80f6860 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.979+0000 7fd5a77fe700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f6860 0x7fd5a810c1b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fd59c0026d0 tx=0x7fd59c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.980+0000 7fd5a4ff9700 1 -- 172.21.15.46:0/1972896418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59c009320 con 0x7fd5a80f6860 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.980+0000 7fd5a4ff9700 1 -- 172.21.15.46:0/1972896418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd59c00f040 con 0x7fd5a80f6860 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.980+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a810c760 con 0x7fd5a80f6860 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.980+0000 7fd5a4ff9700 1 -- 172.21.15.46:0/1972896418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59c0147e0 con 0x7fd5a80f6860 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.980+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a810ccf0 con 0x7fd5a80f6860 2024-01-21T08:18:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.981+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a8045ad0 con 0x7fd5a80f6860 2024-01-21T08:18:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.981+0000 7fd5a4ff9700 1 -- 172.21.15.46:0/1972896418 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd59c009480 con 0x7fd5a80f6860 2024-01-21T08:18:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.981+0000 7fd5a4ff9700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd59005c2b0 0x7fd59005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.982+0000 7fd5a4ff9700 1 -- 172.21.15.46:0/1972896418 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4235+0+0 (secure 0 0 0) 0x7fd59c076030 con 0x7fd5a80f6860 2024-01-21T08:18:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.982+0000 7fd5a7fff700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd59005c2b0 0x7fd59005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.982+0000 7fd5a7fff700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd59005c2b0 0x7fd59005e770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fd5a8060040 tx=0x7fd598009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:12.986+0000 7fd5a4ff9700 1 -- 172.21.15.46:0/1972896418 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd59c0491f0 con 0x7fd5a80f6860 2024-01-21T08:18:13.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.144+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd5a80fde50 con 0x7fd5a80f6860 2024-01-21T08:18:13.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.145+0000 7fd5a4ff9700 1 -- 172.21.15.46:0/1972896418 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v32) v1 ==== 74+0+130 (secure 0 0 0) 0x7fd59c04c810 con 0x7fd5a80f6860 2024-01-21T08:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd59005c2b0 msgr2=0x7fd59005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd59005c2b0 0x7fd59005e770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fd5a8060040 tx=0x7fd598009b30).stop 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f6860 msgr2=0x7fd5a810c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f6860 0x7fd5a810c1b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fd59c0026d0 tx=0x7fd59c009de0).stop 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 shutdown_connections 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd59005c2b0 0x7fd59005e770 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5a80f55c0 0x7fd5a8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 --2- 172.21.15.46:0/1972896418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5a80f6860 0x7fd5a810c1b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 >> 172.21.15.46:0/1972896418 conn(0x7fd5a80f0d40 msgr2=0x7fd5a80fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:13.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.146+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 shutdown_connections 2024-01-21T08:18:13.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:13.147+0000 7fd5ae9a3700 1 -- 172.21.15.46:0/1972896418 wait complete. 2024-01-21T08:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":32,"num_osds":6,"num_up_osds":5,"osd_up_since":1705825083,"num_in_osds":6,"osd_in_since":1705825082,"num_remapped_pgs":0} 2024-01-21T08:18:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:13.146567+0000 mon.smithi046 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.46:0/1972896418' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-21T08:18:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:13.146567+0000 mon.smithi046 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.46:0/1972896418' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-21T08:18:14.531 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd stat -f json 2024-01-21T08:18:15.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:14.092960+0000 mgr.smithi046.lmxnwc (mgr.14164) 434 : cluster [DBG] pgmap v409: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:15.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:15.294049+0000 mon.smithi078 (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-21T08:18:15.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:15.294414+0000 mon.smithi046 (mon.0) 468 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-21T08:18:15.741 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:18:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5[67357]: debug 2024-01-21T08:18:15.289+0000 7f55da08d080 -1 osd.5 0 log_to_monitors {default=true} 2024-01-21T08:18:15.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:14.092960+0000 mgr.smithi046.lmxnwc (mgr.14164) 434 : cluster [DBG] pgmap v409: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:15.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:15.294049+0000 mon.smithi078 (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-21T08:18:15.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:15.294414+0000 mon.smithi046 (mon.0) 468 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-21T08:18:15.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 -- 172.21.15.46:0/3421804208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 msgr2=0x7f65400f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:15.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 --2- 172.21.15.46:0/3421804208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f65400f6b00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f653c004660 tx=0x7f653c009b30).stop 2024-01-21T08:18:15.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 -- 172.21.15.46:0/3421804208 shutdown_connections 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 --2- 172.21.15.46:0/3421804208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65400f7980 0x7f65400f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 --2- 172.21.15.46:0/3421804208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f65400f6b00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 -- 172.21.15.46:0/3421804208 >> 172.21.15.46:0/3421804208 conn(0x7f65400f1e40 msgr2=0x7f65400f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 -- 172.21.15.46:0/3421804208 shutdown_connections 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.749+0000 7f6547e22700 1 -- 172.21.15.46:0/3421804208 wait complete. 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6547e22700 1 Processor -- start 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6547e22700 1 -- start start 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6547e22700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f6540106f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6547e22700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65400f7980 0x7f6540107470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6547e22700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540107a20 con 0x7f65400f66e0 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6547e22700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540107b60 con 0x7f65400f7980 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6545bbe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f6540106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6545bbe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f6540106f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40818/0 (socket says 172.21.15.46:40818) 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6545bbe700 1 -- 172.21.15.46:0/2162294048 learned_addr learned my addr 172.21.15.46:0/2162294048 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f65453bd700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65400f7980 0x7f6540107470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:15.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6545bbe700 1 -- 172.21.15.46:0/2162294048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65400f7980 msgr2=0x7f6540107470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6545bbe700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65400f7980 0x7f6540107470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.750+0000 7f6545bbe700 1 -- 172.21.15.46:0/2162294048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c005040 con 0x7f65400f66e0 2024-01-21T08:18:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.751+0000 7f6545bbe700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f6540106f80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f653c000c00 tx=0x7f653c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.751+0000 7f65367fc700 1 -- 172.21.15.46:0/2162294048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f653c019070 con 0x7f65400f66e0 2024-01-21T08:18:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.751+0000 7f65367fc700 1 -- 172.21.15.46:0/2162294048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f653c007e70 con 0x7f65400f66e0 2024-01-21T08:18:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.751+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65401084a0 con 0x7f65400f66e0 2024-01-21T08:18:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.751+0000 7f65367fc700 1 -- 172.21.15.46:0/2162294048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f653c014db0 con 0x7f65400f66e0 2024-01-21T08:18:15.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.751+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6540108a00 con 0x7f65400f66e0 2024-01-21T08:18:15.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.752+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6540045ad0 con 0x7f65400f66e0 2024-01-21T08:18:15.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.752+0000 7f65367fc700 1 -- 172.21.15.46:0/2162294048 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f653c025070 con 0x7f65400f66e0 2024-01-21T08:18:15.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.753+0000 7f65367fc700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f652c05c2b0 0x7f652c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:15.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.753+0000 7f65367fc700 1 -- 172.21.15.46:0/2162294048 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4256+0+0 (secure 0 0 0) 0x7f653c076910 con 0x7f65400f66e0 2024-01-21T08:18:15.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.753+0000 7f65453bd700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f652c05c2b0 0x7f652c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:15.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.753+0000 7f65453bd700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f652c05c2b0 0x7f652c05e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6530007f80 tx=0x7f653000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:15.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.757+0000 7f65367fc700 1 -- 172.21.15.46:0/2162294048 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f653c049ac0 con 0x7f65400f66e0 2024-01-21T08:18:15.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.918+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f65400feff0 con 0x7f65400f66e0 2024-01-21T08:18:15.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.918+0000 7f65367fc700 1 -- 172.21.15.46:0/2162294048 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v33) v1 ==== 74+0+130 (secure 0 0 0) 0x7f653c04d0e0 con 0x7f65400f66e0 2024-01-21T08:18:15.920 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.919+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f652c05c2b0 msgr2=0x7f652c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.919+0000 7f6547e22700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f652c05c2b0 0x7f652c05e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6530007f80 tx=0x7f653000b040).stop 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.919+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 msgr2=0x7f6540106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.919+0000 7f6547e22700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f6540106f80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f653c000c00 tx=0x7f653c00b040).stop 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.920+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 shutdown_connections 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.920+0000 7f6547e22700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f652c05c2b0 0x7f652c05e770 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.920+0000 7f6547e22700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65400f66e0 0x7f6540106f80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.920+0000 7f6547e22700 1 --2- 172.21.15.46:0/2162294048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65400f7980 0x7f6540107470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.920+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 >> 172.21.15.46:0/2162294048 conn(0x7f65400f1e40 msgr2=0x7f65400fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:15.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.920+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 shutdown_connections 2024-01-21T08:18:15.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:15.920+0000 7f6547e22700 1 -- 172.21.15.46:0/2162294048 wait complete. 2024-01-21T08:18:16.280 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":33,"num_osds":6,"num_up_osds":5,"osd_up_since":1705825083,"num_in_osds":6,"osd_in_since":1705825082,"num_remapped_pgs":0} 2024-01-21T08:18:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:15.437360+0000 mon.smithi046 (mon.0) 469 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-21T08:18:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:15.437409+0000 mon.smithi046 (mon.0) 470 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-01-21T08:18:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:15.437570+0000 mon.smithi046 (mon.0) 471 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:15.437955+0000 mon.smithi078 (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:15.438248+0000 mon.smithi046 (mon.0) 472 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:15.919896+0000 mon.smithi046 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.46:0/2162294048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-21T08:18:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:16.093440+0000 mgr.smithi046.lmxnwc (mgr.14164) 435 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:16.741 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5[67357]: debug 2024-01-21T08:18:16.442+0000 7f55d05fe700 -1 osd.5 0 waiting for initial osdmap 2024-01-21T08:18:16.742 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5[67357]: debug 2024-01-21T08:18:16.452+0000 7f55cad76700 -1 osd.5 34 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-21T08:18:16.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:15.437360+0000 mon.smithi046 (mon.0) 469 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-21T08:18:16.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:15.437409+0000 mon.smithi046 (mon.0) 470 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-01-21T08:18:16.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:15.437570+0000 mon.smithi046 (mon.0) 471 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:16.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:15.437955+0000 mon.smithi078 (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:16.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:15.438248+0000 mon.smithi046 (mon.0) 472 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-01-21T08:18:16.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:15.919896+0000 mon.smithi046 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.46:0/2162294048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-21T08:18:16.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:16.093440+0000 mgr.smithi046.lmxnwc (mgr.14164) 435 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:17.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd stat -f json 2024-01-21T08:18:17.624 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:16.438634+0000 mon.smithi046 (mon.0) 474 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-01-21T08:18:17.624 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:16.438733+0000 mon.smithi046 (mon.0) 475 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-01-21T08:18:17.624 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:16.439102+0000 mon.smithi046 (mon.0) 476 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:17.624 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:16.443195+0000 mon.smithi046 (mon.0) 477 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:16.438634+0000 mon.smithi046 (mon.0) 474 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-01-21T08:18:17.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:16.438733+0000 mon.smithi046 (mon.0) 475 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-01-21T08:18:17.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:16.439102+0000 mon.smithi046 (mon.0) 476 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:17.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:16.443195+0000 mon.smithi046 (mon.0) 477 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 -- 172.21.15.46:0/1146299361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 msgr2=0x7f82c80f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:18.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 --2- 172.21.15.46:0/1146299361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c80f4c20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f82b8004660 tx=0x7f82b8009b30).stop 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 -- 172.21.15.46:0/1146299361 shutdown_connections 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 --2- 172.21.15.46:0/1146299361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c80f51a0 0x7f82c80f7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 --2- 172.21.15.46:0/1146299361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c80f4c20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 -- 172.21.15.46:0/1146299361 >> 172.21.15.46:0/1146299361 conn(0x7f82c80f0320 msgr2=0x7f82c80f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 -- 172.21.15.46:0/1146299361 shutdown_connections 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.502+0000 7f82cd104700 1 -- 172.21.15.46:0/1146299361 wait complete. 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82cd104700 1 Processor -- start 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82cd104700 1 -- start start 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82cd104700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c8106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82cd104700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c80f51a0 0x7f82c8107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82cd104700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c81079d0 con 0x7f82c80f4800 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82cd104700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c8108140 con 0x7f82c80f51a0 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82c7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c8106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:18.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82c7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c8106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40836/0 (socket says 172.21.15.46:40836) 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.503+0000 7f82c7fff700 1 -- 172.21.15.46:0/3197936152 learned_addr learned my addr 172.21.15.46:0/3197936152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82c7fff700 1 -- 172.21.15.46:0/3197936152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c80f51a0 msgr2=0x7f82c8107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82c7fff700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c80f51a0 0x7f82c8107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82c7fff700 1 -- 172.21.15.46:0/3197936152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8005040 con 0x7f82c80f4800 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82c7fff700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c8106ea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f82b8007a70 tx=0x7f82b8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82c4ff9700 1 -- 172.21.15.46:0/3197936152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b8018070 con 0x7f82c80f4800 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c81083c0 con 0x7f82c80f4800 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82c4ff9700 1 -- 172.21.15.46:0/3197936152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82b8009700 con 0x7f82c80f4800 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82c4ff9700 1 -- 172.21.15.46:0/3197936152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b801c620 con 0x7f82c80f4800 2024-01-21T08:18:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.504+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c8108920 con 0x7f82c80f4800 2024-01-21T08:18:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.505+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c8059100 con 0x7f82c80f4800 2024-01-21T08:18:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.505+0000 7f82c4ff9700 1 -- 172.21.15.46:0/3197936152 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f82b801d420 con 0x7f82c80f4800 2024-01-21T08:18:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.506+0000 7f82c4ff9700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f82b005c2f0 0x7f82b005e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.506+0000 7f82c4ff9700 1 -- 172.21.15.46:0/3197936152 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f82b8077a70 con 0x7f82c80f4800 2024-01-21T08:18:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.506+0000 7f82c77fe700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f82b005c2f0 0x7f82b005e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:18.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.506+0000 7f82c77fe700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f82b005c2f0 0x7f82b005e7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f82c80f55e0 tx=0x7f82bc006ab0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:18.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.510+0000 7f82c4ff9700 1 -- 172.21.15.46:0/3197936152 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82b804aa70 con 0x7f82c80f4800 2024-01-21T08:18:18.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.669+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f82c80fed90 con 0x7f82c80f4800 2024-01-21T08:18:18.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.669+0000 7f82c4ff9700 1 -- 172.21.15.46:0/3197936152 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v36) v1 ==== 74+0+130 (secure 0 0 0) 0x7f82b8077e60 con 0x7f82c80f4800 2024-01-21T08:18:18.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:18.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f82b005c2f0 msgr2=0x7f82b005e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:18.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f82b005c2f0 0x7f82b005e7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f82c80f55e0 tx=0x7f82bc006ab0).stop 2024-01-21T08:18:18.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 msgr2=0x7f82c8106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c8106ea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f82b8007a70 tx=0x7f82b8009b30).stop 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 shutdown_connections 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f82b005c2f0 0x7f82b005e7b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c80f4800 0x7f82c8106ea0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 --2- 172.21.15.46:0/3197936152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c80f51a0 0x7f82c8107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 >> 172.21.15.46:0/3197936152 conn(0x7f82c80f0320 msgr2=0x7f82c80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 shutdown_connections 2024-01-21T08:18:18.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:18.671+0000 7f82cd104700 1 -- 172.21.15.46:0/3197936152 wait complete. 2024-01-21T08:18:18.698 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:16.251577+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:18:18.698 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:16.251679+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:18:18.698 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:17.445088+0000 mon.smithi046 (mon.0) 478 : cluster [INF] osd.5 [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] boot 2024-01-21T08:18:18.698 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:17.445142+0000 mon.smithi046 (mon.0) 479 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2024-01-21T08:18:18.698 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:17.445608+0000 mon.smithi046 (mon.0) 480 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:18.698 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:18.093927+0000 mgr.smithi046.lmxnwc (mgr.14164) 436 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:18.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:16.251577+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-21T08:18:18.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:16.251679+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-21T08:18:18.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:17.445088+0000 mon.smithi046 (mon.0) 478 : cluster [INF] osd.5 [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] boot 2024-01-21T08:18:18.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:17.445142+0000 mon.smithi046 (mon.0) 479 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2024-01-21T08:18:18.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:17.445608+0000 mon.smithi046 (mon.0) 480 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-21T08:18:18.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:18.093927+0000 mgr.smithi046.lmxnwc (mgr.14164) 436 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-21T08:18:19.054 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"num_osds":6,"num_up_osds":6,"osd_up_since":1705825097,"num_in_osds":6,"osd_in_since":1705825082,"num_remapped_pgs":0} 2024-01-21T08:18:19.055 INFO:tasks.cephadm:Setting up client nodes... 2024-01-21T08:18:19.055 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-21T08:18:19.709 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:18.454093+0000 mon.smithi046 (mon.0) 481 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-01-21T08:18:19.709 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:18.671187+0000 mon.smithi046 (mon.0) 482 : audit [DBG] from='client.? 172.21.15.46:0/3197936152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-21T08:18:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:18.454093+0000 mon.smithi046 (mon.0) 481 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-01-21T08:18:19.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:18.671187+0000 mon.smithi046 (mon.0) 482 : audit [DBG] from='client.? 172.21.15.46:0/3197936152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-21T08:18:20.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.213+0000 7f8d0b437700 1 -- 172.21.15.46:0/1674847762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f6850 msgr2=0x7f8d040f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:20.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.213+0000 7f8d0b437700 1 --2- 172.21.15.46:0/1674847762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f6850 0x7f8d040f6cb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f8cf4004660 tx=0x7f8cf4009b30).stop 2024-01-21T08:18:20.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.213+0000 7f8d0b437700 1 -- 172.21.15.46:0/1674847762 shutdown_connections 2024-01-21T08:18:20.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.213+0000 7f8d0b437700 1 --2- 172.21.15.46:0/1674847762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f6850 0x7f8d040f6cb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:20.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.213+0000 7f8d0b437700 1 --2- 172.21.15.46:0/1674847762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d040f55b0 0x7f8d040f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:20.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.213+0000 7f8d0b437700 1 -- 172.21.15.46:0/1674847762 >> 172.21.15.46:0/1674847762 conn(0x7f8d040f0d10 msgr2=0x7f8d040f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:20.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.213+0000 7f8d0b437700 1 -- 172.21.15.46:0/1674847762 shutdown_connections 2024-01-21T08:18:20.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d0b437700 1 -- 172.21.15.46:0/1674847762 wait complete. 2024-01-21T08:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d0b437700 1 Processor -- start 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d0b437700 1 -- start start 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d0b437700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f55b0 0x7f8d04105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d0b437700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d040f6850 0x7f8d041063b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d0b437700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d041069f0 con 0x7f8d040f55b0 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d0b437700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d041072a0 con 0x7f8d040f6850 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d091d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f55b0 0x7f8d04105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d091d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f55b0 0x7f8d04105ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40852/0 (socket says 172.21.15.46:40852) 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.214+0000 7f8d091d3700 1 -- 172.21.15.46:0/2255052978 learned_addr learned my addr 172.21.15.46:0/2255052978 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8d089d2700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d040f6850 0x7f8d041063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8d091d3700 1 -- 172.21.15.46:0/2255052978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d040f6850 msgr2=0x7f8d041063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8d091d3700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d040f6850 0x7f8d041063b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8d091d3700 1 -- 172.21.15.46:0/2255052978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf4005040 con 0x7f8d040f55b0 2024-01-21T08:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8d091d3700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f55b0 0x7f8d04105ec0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f8d000093f0 tx=0x7f8d00006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8cf9ffb700 1 -- 172.21.15.46:0/2255052978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d00009ea0 con 0x7f8d040f55b0 2024-01-21T08:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8cf9ffb700 1 -- 172.21.15.46:0/2255052978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d00013af0 con 0x7f8d040f55b0 2024-01-21T08:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d04107580 con 0x7f8d040f55b0 2024-01-21T08:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8cf9ffb700 1 -- 172.21.15.46:0/2255052978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d0000dad0 con 0x7f8d040f55b0 2024-01-21T08:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.215+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d04107b40 con 0x7f8d040f55b0 2024-01-21T08:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.216+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d040f49a0 con 0x7f8d040f55b0 2024-01-21T08:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.217+0000 7f8cf9ffb700 1 -- 172.21.15.46:0/2255052978 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8d0001a060 con 0x7f8d040f55b0 2024-01-21T08:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.217+0000 7f8cf9ffb700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8cf005c2b0 0x7f8cf005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.217+0000 7f8cf9ffb700 1 -- 172.21.15.46:0/2255052978 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8d00075bd0 con 0x7f8d040f55b0 2024-01-21T08:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.217+0000 7f8d089d2700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8cf005c2b0 0x7f8cf005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.217+0000 7f8d089d2700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8cf005c2b0 0x7f8cf005e770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8cf4013580 tx=0x7f8cf4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.221+0000 7f8cf9ffb700 1 -- 172.21.15.46:0/2255052978 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d00048bd0 con 0x7f8d040f55b0 2024-01-21T08:18:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.436+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f8d04045ad0 con 0x7f8d040f55b0 2024-01-21T08:18:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.440+0000 7f8cf9ffb700 1 -- 172.21.15.46:0/2255052978 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f8d0004c1f0 con 0x7f8d040f55b0 2024-01-21T08:18:20.443 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-01-21T08:18:20.443 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBM06xlw+IlGhAAJ5ZXtaknW7SeONxlx2DIJA== 2024-01-21T08:18:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8cf005c2b0 msgr2=0x7f8cf005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8cf005c2b0 0x7f8cf005e770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8cf4013580 tx=0x7f8cf4009b30).stop 2024-01-21T08:18:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f55b0 msgr2=0x7f8d04105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f55b0 0x7f8d04105ec0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f8d000093f0 tx=0x7f8d00006ec0).stop 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 shutdown_connections 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f8cf005c2b0 0x7f8cf005e770 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d040f55b0 0x7f8d04105ec0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0x7f8d000093f0 tx=0x7f8d00006ec0).stop 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 --2- 172.21.15.46:0/2255052978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d040f6850 0x7f8d041063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 >> 172.21.15.46:0/2255052978 conn(0x7f8d040f0d10 msgr2=0x7f8d040fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 shutdown_connections 2024-01-21T08:18:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:20.442+0000 7f8d0b437700 1 -- 172.21.15.46:0/2255052978 wait complete. 2024-01-21T08:18:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:20.094345+0000 mgr.smithi046.lmxnwc (mgr.14164) 437 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:20.438526+0000 mon.smithi046 (mon.0) 483 : audit [INF] from='client.? 172.21.15.46:0/2255052978' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-21T08:18:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:20.441922+0000 mon.smithi046 (mon.0) 484 : audit [INF] from='client.? 172.21.15.46:0/2255052978' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-21T08:18:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:20.094345+0000 mgr.smithi046.lmxnwc (mgr.14164) 437 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:20.438526+0000 mon.smithi046 (mon.0) 483 : audit [INF] from='client.? 172.21.15.46:0/2255052978' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-21T08:18:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:20.441922+0000 mon.smithi046 (mon.0) 484 : audit [INF] from='client.? 172.21.15.46:0/2255052978' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-21T08:18:20.813 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-21T08:18:20.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-01-21T08:18:20.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-21T08:18:20.853 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-21T08:18:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:20.758705+0000 mon.smithi046 (mon.0) 485 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:18:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:20.758910+0000 mon.smithi046 (mon.0) 486 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:18:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:20.758705+0000 mon.smithi046 (mon.0) 485 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:18:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:20.758910+0000 mon.smithi046 (mon.0) 486 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:18:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.035+0000 7ff4a03f2700 1 -- 172.21.15.78:0/2708085935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f55b0 msgr2=0x7ff4980f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.035+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/2708085935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f55b0 0x7ff4980f59d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff494004660 tx=0x7ff494009b30).stop 2024-01-21T08:18:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 -- 172.21.15.78:0/2708085935 shutdown_connections 2024-01-21T08:18:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/2708085935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4980f6850 0x7ff4980f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/2708085935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f55b0 0x7ff4980f59d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 -- 172.21.15.78:0/2708085935 >> 172.21.15.78:0/2708085935 conn(0x7ff4980f0d10 msgr2=0x7ff4980f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 -- 172.21.15.78:0/2708085935 shutdown_connections 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 -- 172.21.15.78:0/2708085935 wait complete. 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 Processor -- start 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 -- start start 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4980f55b0 0x7ff498109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f6850 0x7ff498109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff498109c10 con 0x7ff4980f55b0 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff4a03f2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff498109d50 con 0x7ff4980f6850 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff49d98d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f6850 0x7ff498109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff49d98d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f6850 0x7ff498109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35778/0 (socket says 172.21.15.78:35778) 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff49d98d700 1 -- 172.21.15.78:0/4158177242 learned_addr learned my addr 172.21.15.78:0/4158177242 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.036+0000 7ff49e18e700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4980f55b0 0x7ff498109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff49d98d700 1 -- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4980f55b0 msgr2=0x7ff498109170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff49d98d700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4980f55b0 0x7ff498109170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff49d98d700 1 -- 172.21.15.78:0/4158177242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494005040 con 0x7ff4980f6850 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff49d98d700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f6850 0x7ff498109660 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff48800c530 tx=0x7ff488007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff48effd700 1 -- 172.21.15.78:0/4158177242 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff48800f650 con 0x7ff4980f6850 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff48effd700 1 -- 172.21.15.78:0/4158177242 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff48800fc60 con 0x7ff4980f6850 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff48effd700 1 -- 172.21.15.78:0/4158177242 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff488012c30 con 0x7ff4980f6850 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49810a6f0 con 0x7ff4980f6850 2024-01-21T08:18:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.037+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49810acb0 con 0x7ff4980f6850 2024-01-21T08:18:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.038+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff498045ad0 con 0x7ff4980f6850 2024-01-21T08:18:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.038+0000 7ff48effd700 1 -- 172.21.15.78:0/4158177242 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff488010070 con 0x7ff4980f6850 2024-01-21T08:18:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.039+0000 7ff48effd700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff48405c2b0 0x7ff48405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.039+0000 7ff48effd700 1 -- 172.21.15.78:0/4158177242 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff488075730 con 0x7ff4980f6850 2024-01-21T08:18:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.039+0000 7ff49e18e700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff48405c2b0 0x7ff48405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.039+0000 7ff49e18e700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff48405c2b0 0x7ff48405e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff494004630 tx=0x7ff494009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.042+0000 7ff48effd700 1 -- 172.21.15.78:0/4158177242 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4880452b0 con 0x7ff4980f6850 2024-01-21T08:18:22.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.248+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ff49810b340 con 0x7ff4980f6850 2024-01-21T08:18:22.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.252+0000 7ff48effd700 1 -- 172.21.15.78:0/4158177242 <== mon.1 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (secure 0 0 0) 0x7ff48804c1e0 con 0x7ff4980f6850 2024-01-21T08:18:22.254 INFO:teuthology.orchestra.run.smithi078.stdout:[client.1] 2024-01-21T08:18:22.254 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBO06xlzUrmDhAA6jA1c0z4HA7mxb6FC09THw== 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff48405c2b0 msgr2=0x7ff48405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff48405c2b0 0x7ff48405e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff494004630 tx=0x7ff494009b30).stop 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f6850 msgr2=0x7ff498109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f6850 0x7ff498109660 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff48800c530 tx=0x7ff488007e20).stop 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 shutdown_connections 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff48405c2b0 0x7ff48405e770 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4980f55b0 0x7ff498109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:22.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 --2- 172.21.15.78:0/4158177242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4980f6850 0x7ff498109660 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:22.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 >> 172.21.15.78:0/4158177242 conn(0x7ff4980f0d10 msgr2=0x7ff49805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:22.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 shutdown_connections 2024-01-21T08:18:22.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-01-21T08:18:22.254+0000 7ff4a03f2700 1 -- 172.21.15.78:0/4158177242 wait complete. 2024-01-21T08:18:22.539 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:22.094771+0000 mgr.smithi046.lmxnwc (mgr.14164) 438 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:22.540 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:22.249264+0000 mon.smithi078 (mon.1) 16 : audit [INF] from='client.? 172.21.15.78:0/4158177242' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-21T08:18:22.540 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:22.249811+0000 mon.smithi046 (mon.0) 487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-21T08:18:22.540 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:22.253451+0000 mon.smithi046 (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-21T08:18:22.613 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-01-21T08:18:22.614 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-01-21T08:18:22.614 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-21T08:18:22.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd dump --format=json 2024-01-21T08:18:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:22.094771+0000 mgr.smithi046.lmxnwc (mgr.14164) 438 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:22.249264+0000 mon.smithi078 (mon.1) 16 : audit [INF] from='client.? 172.21.15.78:0/4158177242' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-21T08:18:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:22.249811+0000 mon.smithi046 (mon.0) 487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-21T08:18:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:22.253451+0000 mon.smithi046 (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-21T08:18:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:23.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 -- 172.21.15.46:0/707342841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 msgr2=0x7f44900f6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:23.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 --2- 172.21.15.46:0/707342841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f44900f6070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f4484004660 tx=0x7f4484009b30).stop 2024-01-21T08:18:23.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 -- 172.21.15.46:0/707342841 shutdown_connections 2024-01-21T08:18:23.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 --2- 172.21.15.46:0/707342841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f44900f6070 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:23.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 --2- 172.21.15.46:0/707342841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44900f4de0 0x7f44900f5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:23.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 -- 172.21.15.46:0/707342841 >> 172.21.15.46:0/707342841 conn(0x7f44900f0590 msgr2=0x7f44900f29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:23.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 -- 172.21.15.46:0/707342841 shutdown_connections 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 -- 172.21.15.46:0/707342841 wait complete. 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 Processor -- start 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 -- start start 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.864+0000 7f4494eec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44900f4de0 0x7f4490106040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f4494eec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f4490106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f4494eec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4490106b70 con 0x7f44900f56f0 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f4494eec700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4490107420 con 0x7f44900f4de0 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f4490106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f4490106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37120/0 (socket says 172.21.15.46:37120) 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448dd9b700 1 -- 172.21.15.46:0/2956808342 learned_addr learned my addr 172.21.15.46:0/2956808342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448e59c700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44900f4de0 0x7f4490106040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448dd9b700 1 -- 172.21.15.46:0/2956808342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44900f4de0 msgr2=0x7f4490106040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448dd9b700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44900f4de0 0x7f4490106040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:23.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448dd9b700 1 -- 172.21.15.46:0/2956808342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4484005040 con 0x7f44900f56f0 2024-01-21T08:18:23.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f448dd9b700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f4490106530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f4484009830 tx=0x7f4484009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:23.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f447effd700 1 -- 172.21.15.46:0/2956808342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4484018070 con 0x7f44900f56f0 2024-01-21T08:18:23.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f447effd700 1 -- 172.21.15.46:0/2956808342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4484013410 con 0x7f44900f56f0 2024-01-21T08:18:23.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44901076a0 con 0x7f44900f56f0 2024-01-21T08:18:23.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f447effd700 1 -- 172.21.15.46:0/2956808342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f448401c440 con 0x7f44900f56f0 2024-01-21T08:18:23.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.865+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4490107c00 con 0x7f44900f56f0 2024-01-21T08:18:23.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.866+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4490059100 con 0x7f44900f56f0 2024-01-21T08:18:23.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.867+0000 7f447effd700 1 -- 172.21.15.46:0/2956808342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f448401d420 con 0x7f44900f56f0 2024-01-21T08:18:23.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.867+0000 7f447effd700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f447805c2a0 0x7f447805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:23.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.867+0000 7f447effd700 1 -- 172.21.15.46:0/2956808342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4484077900 con 0x7f44900f56f0 2024-01-21T08:18:23.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.867+0000 7f448e59c700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f447805c2a0 0x7f447805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:23.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.867+0000 7f448e59c700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f447805c2a0 0x7f447805e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f44900f5be0 tx=0x7f4480008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:23.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:23.871+0000 7f447effd700 1 -- 172.21.15.46:0/2956808342 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f448404a920 con 0x7f44900f56f0 2024-01-21T08:18:24.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.032+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4490045ad0 con 0x7f44900f56f0 2024-01-21T08:18:24.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.033+0000 7f447effd700 1 -- 172.21.15.46:0/2956808342 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10876 (secure 0 0 0) 0x7f448404df40 con 0x7f44900f56f0 2024-01-21T08:18:24.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:24.036 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","created":"2024-01-21T08:04:26.122565+0000","modified":"2024-01-21T08:18:18.446739+0000","last_up_change":"2024-01-21T08:18:17.440392+0000","last_in_change":"2024-01-21T08:18:02.974749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-21T08:15:21.520576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b512ed9c-653b-46a7-9b39-560a59ab1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6803","nonce":2710852688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6805","nonce":2710852688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6809","nonce":2710852688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6807","nonce":2710852688}]},"public_addr":"172.21.15.46:6803/2710852688","cluster_addr":"172.21.15.46:6805/2710852688","heartbeat_back_addr":"172.21.15.46:6809/2710852688","heartbeat_front_addr":"172.21.15.46:6807/2710852688","state":["exists","up"]},{"osd":1,"uuid":"82fe182c-88a4-4c1a-a563-a1f281f03bd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6811","nonce":3872661900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6813","nonce":3872661900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6817","nonce":3872661900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6815","nonce":3872661900}]},"public_addr":"172.21.15.46:6811/3872661900","cluster_addr":"172.21.15.46:6813/3872661900","heartbeat_back_addr":"172.21.15.46:6817/3872661900","heartbeat_front_addr":"172.21.15.46:6815/3872661900","state":["exists","up"]},{"osd":2,"uuid":"4715fc0d-a749-4eab-ad3e-1934b586b33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6819","nonce":609261533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6821","nonce":609261533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6825","nonce":609261533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6823","nonce":609261533}]},"public_addr":"172.21.15.46:6819/609261533","cluster_addr":"172.21.15.46:6821/609261533","heartbeat_back_addr":"172.21.15.46:6825/609261533","heartbeat_front_addr":"172.21.15.46:6823/609261533","state":["exists","up"]},{"osd":3,"uuid":"46321964-fb94-4a83-873e-ffe70b11e1c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6801","nonce":1326350774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6803","nonce":1326350774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6807","nonce":1326350774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6805","nonce":1326350774}]},"public_addr":"172.21.15.78:6801/1326350774","cluster_addr":"172.21.15.78:6803/1326350774","heartbeat_back_addr":"172.21.15.78:6807/1326350774","heartbeat_front_addr":"172.21.15.78:6805/1326350774","state":["exists","up"]},{"osd":4,"uuid":"a04900e5-161b-4cb2-be30-fd8513a1c2b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6809","nonce":818610611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6811","nonce":818610611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6815","nonce":818610611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6813","nonce":818610611}]},"public_addr":"172.21.15.78:6809/818610611","cluster_addr":"172.21.15.78:6811/818610611","heartbeat_back_addr":"172.21.15.78:6815/818610611","heartbeat_front_addr":"172.21.15.78:6813/818610611","state":["exists","up"]},{"osd":5,"uuid":"43f2f136-72a2-4e01-901e-e2e244f3cf89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6817","nonce":1959692294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6819","nonce":1959692294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6823","nonce":1959692294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6821","nonce":1959692294}]},"public_addr":"172.21.15.78:6817/1959692294","cluster_addr":"172.21.15.78:6819/1959692294","heartbeat_back_addr":"172.21.15.78:6823/1959692294","heartbeat_front_addr":"172.21.15.78:6821/1959692294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:07.734787+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:21.286559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:34.210889+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:15:26.093705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:18:02.722249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:18:16.251682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4034522265":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6801/1810147681":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6800/1810147681":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6801/2245961460":"2024-01-22T08:05:08.021682+0000","172.21.15.46:6800/2245961460":"2024-01-22T08:05:08.021682+0000","172.21.15.46:0/1549907352":"2024-01-22T08:05:08.021682+0000","172.21.15.46:6800/1991662361":"2024-01-22T08:04:46.418654+0000","172.21.15.46:6801/1991662361":"2024-01-22T08:04:46.418654+0000","172.21.15.46:0/1054687982":"2024-01-22T08:05:20.535551+0000","172.21.15.46:0/3467288224":"2024-01-22T08:04:46.418654+0000","172.21.15.46:0/2734037743":"2024-01-22T08:05:08.021682+0000","172.21.15.46:0/3778057593":"2024-01-22T08:04:46.418654+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T08:18:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.034+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f447805c2a0 msgr2=0x7f447805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.034+0000 7f4494eec700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f447805c2a0 0x7f447805e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f44900f5be0 tx=0x7f4480008040).stop 2024-01-21T08:18:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.034+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 msgr2=0x7f4490106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.034+0000 7f4494eec700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f4490106530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f4484009830 tx=0x7f4484009b30).stop 2024-01-21T08:18:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.035+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 shutdown_connections 2024-01-21T08:18:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.035+0000 7f4494eec700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f447805c2a0 0x7f447805e760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.035+0000 7f4494eec700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44900f4de0 0x7f4490106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.035+0000 7f4494eec700 1 --2- 172.21.15.46:0/2956808342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44900f56f0 0x7f4490106530 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.035+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 >> 172.21.15.46:0/2956808342 conn(0x7f44900f0590 msgr2=0x7f44900fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.035+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 shutdown_connections 2024-01-21T08:18:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:24.035+0000 7f4494eec700 1 -- 172.21.15.46:0/2956808342 wait complete. 2024-01-21T08:18:24.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:24.034380+0000 mon.smithi046 (mon.0) 489 : audit [DBG] from='client.? 172.21.15.46:0/2956808342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-21T08:18:24.414 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-01-21T08:15:21.520576+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2024-01-21T08:18:24.415 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-01-21T08:18:24.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:24.034380+0000 mon.smithi046 (mon.0) 489 : audit [DBG] from='client.? 172.21.15.46:0/2956808342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-21T08:18:24.646 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:25.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3620084608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 msgr2=0x7f3ac40f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:25.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3620084608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac40f6de0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f3ab8004660 tx=0x7f3ab8009b30).stop 2024-01-21T08:18:25.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3620084608 shutdown_connections 2024-01-21T08:18:25.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3620084608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac40f6de0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:25.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3620084608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac40f56e0 0x7f3ac40f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:25.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3620084608 >> 172.21.15.46:0/3620084608 conn(0x7f3ac40f0e20 msgr2=0x7f3ac40f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:25.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3620084608 shutdown_connections 2024-01-21T08:18:25.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.583+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3620084608 wait complete. 2024-01-21T08:18:25.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac97fc700 1 Processor -- start 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac97fc700 1 -- start start 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac97fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac40f56e0 0x7f3ac4109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac97fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac410c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac97fc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac4060920 con 0x7f3ac40f6980 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac97fc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac4060a60 con 0x7f3ac40f56e0 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac410c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac410c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37148/0 (socket says 172.21.15.46:37148) 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.584+0000 7f3ac27fc700 1 -- 172.21.15.46:0/3386328944 learned_addr learned my addr 172.21.15.46:0/3386328944 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3ac2ffd700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac40f56e0 0x7f3ac4109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3ac27fc700 1 -- 172.21.15.46:0/3386328944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac40f56e0 msgr2=0x7f3ac4109dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3ac27fc700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac40f56e0 0x7f3ac4109dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3ac27fc700 1 -- 172.21.15.46:0/3386328944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab8005040 con 0x7f3ac40f6980 2024-01-21T08:18:25.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3ac27fc700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac410c2c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f3ab8009830 tx=0x7f3ab8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:25.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3aab7fe700 1 -- 172.21.15.46:0/3386328944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab8018070 con 0x7f3ac40f6980 2024-01-21T08:18:25.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac410c810 con 0x7f3ac40f6980 2024-01-21T08:18:25.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3aab7fe700 1 -- 172.21.15.46:0/3386328944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ab8013410 con 0x7f3ac40f6980 2024-01-21T08:18:25.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3aab7fe700 1 -- 172.21.15.46:0/3386328944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab801c620 con 0x7f3ac40f6980 2024-01-21T08:18:25.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.585+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac410cd40 con 0x7f3ac40f6980 2024-01-21T08:18:25.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.586+0000 7f3aab7fe700 1 -- 172.21.15.46:0/3386328944 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3ab801c8e0 con 0x7f3ac40f6980 2024-01-21T08:18:25.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.586+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac4059100 con 0x7f3ac40f6980 2024-01-21T08:18:25.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.587+0000 7f3aab7fe700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3aac05c2a0 0x7f3aac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:25.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.587+0000 7f3aab7fe700 1 -- 172.21.15.46:0/3386328944 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3ab8077a60 con 0x7f3ac40f6980 2024-01-21T08:18:25.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.587+0000 7f3ac2ffd700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3aac05c2a0 0x7f3aac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:25.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.587+0000 7f3ac2ffd700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3aac05c2a0 0x7f3aac05e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3ac40605a0 tx=0x7f3ab4009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:25.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.591+0000 7f3aab7fe700 1 -- 172.21.15.46:0/3386328944 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ab804aa00 con 0x7f3ac40f6980 2024-01-21T08:18:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:24.095052+0000 mgr.smithi046.lmxnwc (mgr.14164) 439 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:24.095052+0000 mgr.smithi046.lmxnwc (mgr.14164) 439 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.751+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3ac4045ad0 con 0x7f3ac40f6980 2024-01-21T08:18:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.751+0000 7f3aab7fe700 1 -- 172.21.15.46:0/3386328944 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v36) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3ab804e020 con 0x7f3ac40f6980 2024-01-21T08:18:25.754 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3aac05c2a0 msgr2=0x7f3aac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3aac05c2a0 0x7f3aac05e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3ac40605a0 tx=0x7f3ab4009040).stop 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 msgr2=0x7f3ac410c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac410c2c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f3ab8009830 tx=0x7f3ab8009b30).stop 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 shutdown_connections 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3aac05c2a0 0x7f3aac05e760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac40f56e0 0x7f3ac4109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 --2- 172.21.15.46:0/3386328944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac40f6980 0x7f3ac410c2c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 >> 172.21.15.46:0/3386328944 conn(0x7f3ac40f0e20 msgr2=0x7f3ac40fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 shutdown_connections 2024-01-21T08:18:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:25.753+0000 7f3ac97fc700 1 -- 172.21.15.46:0/3386328944 wait complete. 2024-01-21T08:18:26.129 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-21T08:18:26.129 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-01-21T08:18:26.129 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph mgr dump --format=json 2024-01-21T08:18:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:26.670 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:25.753434+0000 mon.smithi046 (mon.0) 490 : audit [DBG] from='client.? 172.21.15.46:0/3386328944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-01-21T08:18:26.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:25.753434+0000 mon.smithi046 (mon.0) 490 : audit [DBG] from='client.? 172.21.15.46:0/3386328944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.320+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3755925949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 msgr2=0x7ff1280f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.320+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3755925949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff1280f91f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff124004660 tx=0x7ff124009b30).stop 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.320+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3755925949 shutdown_connections 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.320+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3755925949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1280f9770 0x7ff1280fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.320+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3755925949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff1280f91f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.320+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3755925949 >> 172.21.15.46:0/3755925949 conn(0x7ff1280f0c70 msgr2=0x7ff1280f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3755925949 shutdown_connections 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3755925949 wait complete. 2024-01-21T08:18:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 Processor -- start 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 -- start start 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff128106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1280f9770 0x7ff1281073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128107990 con 0x7ff128060870 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12eb9a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128107ad0 con 0x7ff1280f9770 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12db98700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff128106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12db98700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff128106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37168/0 (socket says 172.21.15.46:37168) 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.321+0000 7ff12db98700 1 -- 172.21.15.46:0/3795821382 learned_addr learned my addr 172.21.15.46:0/3795821382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff12d397700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1280f9770 0x7ff1281073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff12db98700 1 -- 172.21.15.46:0/3795821382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1280f9770 msgr2=0x7ff1281073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:27.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff12db98700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1280f9770 0x7ff1281073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:27.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff12db98700 1 -- 172.21.15.46:0/3795821382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff124005040 con 0x7ff128060870 2024-01-21T08:18:27.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff12db98700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff128106ef0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff124009f50 tx=0x7ff124009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:27.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff11e7fc700 1 -- 172.21.15.46:0/3795821382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff124018070 con 0x7ff128060870 2024-01-21T08:18:27.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff128108410 con 0x7ff128060870 2024-01-21T08:18:27.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff11e7fc700 1 -- 172.21.15.46:0/3795821382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff124013410 con 0x7ff128060870 2024-01-21T08:18:27.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff11e7fc700 1 -- 172.21.15.46:0/3795821382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff12401c3c0 con 0x7ff128060870 2024-01-21T08:18:27.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.322+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff128108970 con 0x7ff128060870 2024-01-21T08:18:27.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.323+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff128045ad0 con 0x7ff128060870 2024-01-21T08:18:27.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.323+0000 7ff11e7fc700 1 -- 172.21.15.46:0/3795821382 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff12401d420 con 0x7ff128060870 2024-01-21T08:18:27.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.324+0000 7ff11e7fc700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff11405c2b0 0x7ff11405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:27.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.324+0000 7ff11e7fc700 1 -- 172.21.15.46:0/3795821382 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff124077800 con 0x7ff128060870 2024-01-21T08:18:27.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.324+0000 7ff12d397700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff11405c2b0 0x7ff11405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:27.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.324+0000 7ff12d397700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff11405c2b0 0x7ff11405e770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff128060640 tx=0x7ff11800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.328+0000 7ff11e7fc700 1 -- 172.21.15.46:0/3795821382 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff12404a800 con 0x7ff128060870 2024-01-21T08:18:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.528+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff128027a00 con 0x7ff128060870 2024-01-21T08:18:27.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.531+0000 7ff11e7fc700 1 -- 172.21.15.46:0/3795821382 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v18) v1 ==== 74+0+143495 (secure 0 0 0) 0x7ff12404de20 con 0x7ff128060870 2024-01-21T08:18:27.536 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff11405c2b0 msgr2=0x7ff11405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff11405c2b0 0x7ff11405e770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff128060640 tx=0x7ff11800b040).stop 2024-01-21T08:18:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 msgr2=0x7ff128106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff128106ef0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff124009f50 tx=0x7ff124009b30).stop 2024-01-21T08:18:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 shutdown_connections 2024-01-21T08:18:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff11405c2b0 0x7ff11405e770 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff128060870 0x7ff128106ef0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 --2- 172.21.15.46:0/3795821382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1280f9770 0x7ff1281073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 >> 172.21.15.46:0/3795821382 conn(0x7ff1280f0c70 msgr2=0x7ff1280f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 shutdown_connections 2024-01-21T08:18:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:27.533+0000 7ff12eb9a700 1 -- 172.21.15.46:0/3795821382 wait complete. 2024-01-21T08:18:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:26.095550+0000 mgr.smithi046.lmxnwc (mgr.14164) 440 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:26.095550+0000 mgr.smithi046.lmxnwc (mgr.14164) 440 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:27.940 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"active_gid":14164,"active_name":"smithi046.lmxnwc","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":559524926},{"type":"v1","addr":"172.21.15.46:6801","nonce":559524926}]},"active_addr":"172.21.15.46:6801/559524926","active_change":"2024-01-21T08:05:20.535924+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14384,"name":"smithi078.kcfjgo","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi046.front.sepia.ceph.com:8443/","prometheus":"http://smithi046.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3076160887}]},{"addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":67936841}]}]}} 2024-01-21T08:18:27.945 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-01-21T08:18:27.945 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-01-21T08:18:27.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd dump --format=json 2024-01-21T08:18:28.175 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:28.616 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:27.530466+0000 mon.smithi046 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.46:0/3795821382' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-21T08:18:28.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:27.530466+0000 mon.smithi046 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.46:0/3795821382' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-21T08:18:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 -- 172.21.15.46:0/3020264382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 msgr2=0x7fef180f7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 --2- 172.21.15.46:0/3020264382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef180f7d20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fef0c004660 tx=0x7fef0c009b30).stop 2024-01-21T08:18:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 -- 172.21.15.46:0/3020264382 shutdown_connections 2024-01-21T08:18:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 --2- 172.21.15.46:0/3020264382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef180f7d20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 --2- 172.21.15.46:0/3020264382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef180f6620 0x7fef180f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 -- 172.21.15.46:0/3020264382 >> 172.21.15.46:0/3020264382 conn(0x7fef180f1da0 msgr2=0x7fef180f41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:29.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 -- 172.21.15.46:0/3020264382 shutdown_connections 2024-01-21T08:18:29.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.121+0000 7fef17fff700 1 -- 172.21.15.46:0/3020264382 wait complete. 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef17fff700 1 Processor -- start 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef17fff700 1 -- start start 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef17fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef180f6620 0x7fef18106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef17fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef181073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef17fff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef18107980 con 0x7fef180f78c0 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef17fff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef18107ac0 con 0x7fef180f6620 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef167fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef181073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef167fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef181073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37184/0 (socket says 172.21.15.46:37184) 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.122+0000 7fef167fc700 1 -- 172.21.15.46:0/2070350922 learned_addr learned my addr 172.21.15.46:0/2070350922 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7fef16ffd700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef180f6620 0x7fef18106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7fef167fc700 1 -- 172.21.15.46:0/2070350922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef180f6620 msgr2=0x7fef18106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7fef167fc700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef180f6620 0x7fef18106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7fef167fc700 1 -- 172.21.15.46:0/2070350922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c005040 con 0x7fef180f78c0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7fef167fc700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef181073d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fef0c009f50 tx=0x7fef0c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7feeff7fe700 1 -- 172.21.15.46:0/2070350922 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef0c018070 con 0x7fef180f78c0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7feeff7fe700 1 -- 172.21.15.46:0/2070350922 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef0c013410 con 0x7fef180f78c0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef18108400 con 0x7fef180f78c0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7feeff7fe700 1 -- 172.21.15.46:0/2070350922 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef0c01c3c0 con 0x7fef180f78c0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.123+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef18108960 con 0x7fef180f78c0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.124+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef18045ad0 con 0x7fef180f78c0 2024-01-21T08:18:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.125+0000 7feeff7fe700 1 -- 172.21.15.46:0/2070350922 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fef0c01d420 con 0x7fef180f78c0 2024-01-21T08:18:29.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.125+0000 7feeff7fe700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fef0005c2b0 0x7fef0005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:29.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.125+0000 7feeff7fe700 1 -- 172.21.15.46:0/2070350922 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fef0c077730 con 0x7fef180f78c0 2024-01-21T08:18:29.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.125+0000 7fef16ffd700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fef0005c2b0 0x7fef0005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:29.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.125+0000 7fef16ffd700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fef0005c2b0 0x7fef0005e770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fef180f7720 tx=0x7fef0800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:29.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.129+0000 7feeff7fe700 1 -- 172.21.15.46:0/2070350922 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef0c04a7b0 con 0x7fef180f78c0 2024-01-21T08:18:29.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.287+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef18027a00 con 0x7fef180f78c0 2024-01-21T08:18:29.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.288+0000 7feeff7fe700 1 -- 172.21.15.46:0/2070350922 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10876 (secure 0 0 0) 0x7fef0c04ddd0 con 0x7fef180f78c0 2024-01-21T08:18:29.291 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:29.291 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","created":"2024-01-21T08:04:26.122565+0000","modified":"2024-01-21T08:18:18.446739+0000","last_up_change":"2024-01-21T08:18:17.440392+0000","last_in_change":"2024-01-21T08:18:02.974749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-21T08:15:21.520576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b512ed9c-653b-46a7-9b39-560a59ab1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6803","nonce":2710852688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6805","nonce":2710852688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6809","nonce":2710852688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6807","nonce":2710852688}]},"public_addr":"172.21.15.46:6803/2710852688","cluster_addr":"172.21.15.46:6805/2710852688","heartbeat_back_addr":"172.21.15.46:6809/2710852688","heartbeat_front_addr":"172.21.15.46:6807/2710852688","state":["exists","up"]},{"osd":1,"uuid":"82fe182c-88a4-4c1a-a563-a1f281f03bd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6811","nonce":3872661900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6813","nonce":3872661900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6817","nonce":3872661900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6815","nonce":3872661900}]},"public_addr":"172.21.15.46:6811/3872661900","cluster_addr":"172.21.15.46:6813/3872661900","heartbeat_back_addr":"172.21.15.46:6817/3872661900","heartbeat_front_addr":"172.21.15.46:6815/3872661900","state":["exists","up"]},{"osd":2,"uuid":"4715fc0d-a749-4eab-ad3e-1934b586b33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6819","nonce":609261533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6821","nonce":609261533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6825","nonce":609261533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6823","nonce":609261533}]},"public_addr":"172.21.15.46:6819/609261533","cluster_addr":"172.21.15.46:6821/609261533","heartbeat_back_addr":"172.21.15.46:6825/609261533","heartbeat_front_addr":"172.21.15.46:6823/609261533","state":["exists","up"]},{"osd":3,"uuid":"46321964-fb94-4a83-873e-ffe70b11e1c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6801","nonce":1326350774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6803","nonce":1326350774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6807","nonce":1326350774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6805","nonce":1326350774}]},"public_addr":"172.21.15.78:6801/1326350774","cluster_addr":"172.21.15.78:6803/1326350774","heartbeat_back_addr":"172.21.15.78:6807/1326350774","heartbeat_front_addr":"172.21.15.78:6805/1326350774","state":["exists","up"]},{"osd":4,"uuid":"a04900e5-161b-4cb2-be30-fd8513a1c2b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6809","nonce":818610611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6811","nonce":818610611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6815","nonce":818610611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6813","nonce":818610611}]},"public_addr":"172.21.15.78:6809/818610611","cluster_addr":"172.21.15.78:6811/818610611","heartbeat_back_addr":"172.21.15.78:6815/818610611","heartbeat_front_addr":"172.21.15.78:6813/818610611","state":["exists","up"]},{"osd":5,"uuid":"43f2f136-72a2-4e01-901e-e2e244f3cf89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6817","nonce":1959692294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6819","nonce":1959692294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6823","nonce":1959692294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6821","nonce":1959692294}]},"public_addr":"172.21.15.78:6817/1959692294","cluster_addr":"172.21.15.78:6819/1959692294","heartbeat_back_addr":"172.21.15.78:6823/1959692294","heartbeat_front_addr":"172.21.15.78:6821/1959692294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:07.734787+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:21.286559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:34.210889+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:15:26.093705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:18:02.722249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:18:16.251682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4034522265":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6801/1810147681":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6800/1810147681":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6801/2245961460":"2024-01-22T08:05:08.021682+0000","172.21.15.46:6800/2245961460":"2024-01-22T08:05:08.021682+0000","172.21.15.46:0/1549907352":"2024-01-22T08:05:08.021682+0000","172.21.15.46:6800/1991662361":"2024-01-22T08:04:46.418654+0000","172.21.15.46:6801/1991662361":"2024-01-22T08:04:46.418654+0000","172.21.15.46:0/1054687982":"2024-01-22T08:05:20.535551+0000","172.21.15.46:0/3467288224":"2024-01-22T08:04:46.418654+0000","172.21.15.46:0/2734037743":"2024-01-22T08:05:08.021682+0000","172.21.15.46:0/3778057593":"2024-01-22T08:04:46.418654+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T08:18:29.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.289+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fef0005c2b0 msgr2=0x7fef0005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:29.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.289+0000 7fef17fff700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fef0005c2b0 0x7fef0005e770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fef180f7720 tx=0x7fef0800b040).stop 2024-01-21T08:18:29.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 msgr2=0x7fef181073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:29.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef181073d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fef0c009f50 tx=0x7fef0c009b30).stop 2024-01-21T08:18:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 shutdown_connections 2024-01-21T08:18:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fef0005c2b0 0x7fef0005e770 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef180f6620 0x7fef18106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 --2- 172.21.15.46:0/2070350922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef180f78c0 0x7fef181073d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 >> 172.21.15.46:0/2070350922 conn(0x7fef180f1da0 msgr2=0x7fef180fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 shutdown_connections 2024-01-21T08:18:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:29.290+0000 7fef17fff700 1 -- 172.21.15.46:0/2070350922 wait complete. 2024-01-21T08:18:29.586 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:28.095938+0000 mgr.smithi046.lmxnwc (mgr.14164) 441 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:29.586 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:29.289527+0000 mon.smithi046 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.46:0/2070350922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-21T08:18:29.670 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-01-21T08:18:29.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd dump --format=json 2024-01-21T08:18:29.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:28.095938+0000 mgr.smithi046.lmxnwc (mgr.14164) 441 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:29.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:29.289527+0000 mon.smithi046 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.46:0/2070350922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-21T08:18:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:30.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/3910544804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 msgr2=0x7f1c680fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:30.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/3910544804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c680fbd70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f1c5c004660 tx=0x7f1c5c009b30).stop 2024-01-21T08:18:30.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/3910544804 shutdown_connections 2024-01-21T08:18:30.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/3910544804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c680fbd70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:30.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/3910544804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c68060870 0x7f1c680f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:30.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/3910544804 >> 172.21.15.46:0/3910544804 conn(0x7f1c680f0e20 msgr2=0x7f1c680f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:30.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/3910544804 shutdown_connections 2024-01-21T08:18:30.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.899+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/3910544804 wait complete. 2024-01-21T08:18:30.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.900+0000 7f1c6e7d8700 1 Processor -- start 2024-01-21T08:18:30.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.900+0000 7f1c6e7d8700 1 -- start start 2024-01-21T08:18:30.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c6e7d8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c68060870 0x7f1c68107100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:30.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c6e7d8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c681075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c6e7d8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c68107ba0 con 0x7f1c680f9920 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c6e7d8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c68107ce0 con 0x7f1c68060870 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c677fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c681075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c677fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c681075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40378/0 (socket says 172.21.15.46:40378) 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c677fe700 1 -- 172.21.15.46:0/682292270 learned_addr learned my addr 172.21.15.46:0/682292270 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c67fff700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c68060870 0x7f1c68107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c677fe700 1 -- 172.21.15.46:0/682292270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c68060870 msgr2=0x7f1c68107100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c677fe700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c68060870 0x7f1c68107100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c677fe700 1 -- 172.21.15.46:0/682292270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c005040 con 0x7f1c680f9920 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c677fe700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c681075f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f1c5c009830 tx=0x7f1c5c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c64ff9700 1 -- 172.21.15.46:0/682292270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c5c018070 con 0x7f1c680f9920 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c64ff9700 1 -- 172.21.15.46:0/682292270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c5c007ba0 con 0x7f1c680f9920 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c68108620 con 0x7f1c680f9920 2024-01-21T08:18:30.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c64ff9700 1 -- 172.21.15.46:0/682292270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c5c01c3c0 con 0x7f1c680f9920 2024-01-21T08:18:30.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.901+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c68108b80 con 0x7f1c680f9920 2024-01-21T08:18:30.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.902+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c68045ad0 con 0x7f1c680f9920 2024-01-21T08:18:30.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.903+0000 7f1c64ff9700 1 -- 172.21.15.46:0/682292270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1c5c01c520 con 0x7f1c680f9920 2024-01-21T08:18:30.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.903+0000 7f1c64ff9700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c5005c2a0 0x7f1c5005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:30.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.903+0000 7f1c64ff9700 1 -- 172.21.15.46:0/682292270 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1c5c077790 con 0x7f1c680f9920 2024-01-21T08:18:30.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.903+0000 7f1c67fff700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c5005c2a0 0x7f1c5005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:30.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.903+0000 7f1c67fff700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c5005c2a0 0x7f1c5005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1c580096c0 tx=0x7f1c58009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:30.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:30.907+0000 7f1c64ff9700 1 -- 172.21.15.46:0/682292270 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c5c04a790 con 0x7f1c680f9920 2024-01-21T08:18:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.067+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c68027a00 con 0x7f1c680f9920 2024-01-21T08:18:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.068+0000 7f1c64ff9700 1 -- 172.21.15.46:0/682292270 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10876 (secure 0 0 0) 0x7f1c5c04ddb0 con 0x7f1c680f9920 2024-01-21T08:18:31.070 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:31.070 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"fsid":"a17ce476-b833-11ee-95b0-87774f69a715","created":"2024-01-21T08:04:26.122565+0000","modified":"2024-01-21T08:18:18.446739+0000","last_up_change":"2024-01-21T08:18:17.440392+0000","last_in_change":"2024-01-21T08:18:02.974749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-21T08:15:21.520576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b512ed9c-653b-46a7-9b39-560a59ab1015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6803","nonce":2710852688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6805","nonce":2710852688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6809","nonce":2710852688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2710852688},{"type":"v1","addr":"172.21.15.46:6807","nonce":2710852688}]},"public_addr":"172.21.15.46:6803/2710852688","cluster_addr":"172.21.15.46:6805/2710852688","heartbeat_back_addr":"172.21.15.46:6809/2710852688","heartbeat_front_addr":"172.21.15.46:6807/2710852688","state":["exists","up"]},{"osd":1,"uuid":"82fe182c-88a4-4c1a-a563-a1f281f03bd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6811","nonce":3872661900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6813","nonce":3872661900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6817","nonce":3872661900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3872661900},{"type":"v1","addr":"172.21.15.46:6815","nonce":3872661900}]},"public_addr":"172.21.15.46:6811/3872661900","cluster_addr":"172.21.15.46:6813/3872661900","heartbeat_back_addr":"172.21.15.46:6817/3872661900","heartbeat_front_addr":"172.21.15.46:6815/3872661900","state":["exists","up"]},{"osd":2,"uuid":"4715fc0d-a749-4eab-ad3e-1934b586b33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6819","nonce":609261533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6821","nonce":609261533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6825","nonce":609261533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":609261533},{"type":"v1","addr":"172.21.15.46:6823","nonce":609261533}]},"public_addr":"172.21.15.46:6819/609261533","cluster_addr":"172.21.15.46:6821/609261533","heartbeat_back_addr":"172.21.15.46:6825/609261533","heartbeat_front_addr":"172.21.15.46:6823/609261533","state":["exists","up"]},{"osd":3,"uuid":"46321964-fb94-4a83-873e-ffe70b11e1c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6801","nonce":1326350774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6803","nonce":1326350774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6807","nonce":1326350774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1326350774},{"type":"v1","addr":"172.21.15.78:6805","nonce":1326350774}]},"public_addr":"172.21.15.78:6801/1326350774","cluster_addr":"172.21.15.78:6803/1326350774","heartbeat_back_addr":"172.21.15.78:6807/1326350774","heartbeat_front_addr":"172.21.15.78:6805/1326350774","state":["exists","up"]},{"osd":4,"uuid":"a04900e5-161b-4cb2-be30-fd8513a1c2b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6809","nonce":818610611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6811","nonce":818610611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6815","nonce":818610611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":818610611},{"type":"v1","addr":"172.21.15.78:6813","nonce":818610611}]},"public_addr":"172.21.15.78:6809/818610611","cluster_addr":"172.21.15.78:6811/818610611","heartbeat_back_addr":"172.21.15.78:6815/818610611","heartbeat_front_addr":"172.21.15.78:6813/818610611","state":["exists","up"]},{"osd":5,"uuid":"43f2f136-72a2-4e01-901e-e2e244f3cf89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6817","nonce":1959692294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6819","nonce":1959692294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6823","nonce":1959692294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1959692294},{"type":"v1","addr":"172.21.15.78:6821","nonce":1959692294}]},"public_addr":"172.21.15.78:6817/1959692294","cluster_addr":"172.21.15.78:6819/1959692294","heartbeat_back_addr":"172.21.15.78:6823/1959692294","heartbeat_front_addr":"172.21.15.78:6821/1959692294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:07.734787+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:21.286559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:11:34.210889+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:15:26.093705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:18:02.722249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T08:18:16.251682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4034522265":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6801/1810147681":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6800/1810147681":"2024-01-22T08:05:20.535551+0000","172.21.15.46:6801/2245961460":"2024-01-22T08:05:08.021682+0000","172.21.15.46:6800/2245961460":"2024-01-22T08:05:08.021682+0000","172.21.15.46:0/1549907352":"2024-01-22T08:05:08.021682+0000","172.21.15.46:6800/1991662361":"2024-01-22T08:04:46.418654+0000","172.21.15.46:6801/1991662361":"2024-01-22T08:04:46.418654+0000","172.21.15.46:0/1054687982":"2024-01-22T08:05:20.535551+0000","172.21.15.46:0/3467288224":"2024-01-22T08:04:46.418654+0000","172.21.15.46:0/2734037743":"2024-01-22T08:05:08.021682+0000","172.21.15.46:0/3778057593":"2024-01-22T08:04:46.418654+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c5005c2a0 msgr2=0x7f1c5005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c5005c2a0 0x7f1c5005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1c580096c0 tx=0x7f1c58009be0).stop 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 msgr2=0x7f1c681075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c681075f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f1c5c009830 tx=0x7f1c5c009b30).stop 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 shutdown_connections 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f1c5005c2a0 0x7f1c5005e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c68060870 0x7f1c68107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 --2- 172.21.15.46:0/682292270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c680f9920 0x7f1c681075f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 >> 172.21.15.46:0/682292270 conn(0x7f1c680f0e20 msgr2=0x7f1c680f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 shutdown_connections 2024-01-21T08:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:31.069+0000 7f1c6e7d8700 1 -- 172.21.15.46:0/682292270 wait complete. 2024-01-21T08:18:31.445 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-01-21T08:18:31.674 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:30.096284+0000 mgr.smithi046.lmxnwc (mgr.14164) 442 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:31.069161+0000 mon.smithi046 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.46:0/682292270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-21T08:18:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:30.096284+0000 mgr.smithi046.lmxnwc (mgr.14164) 442 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:31.069161+0000 mon.smithi046 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.46:0/682292270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 -- 172.21.15.46:0/2045606203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 msgr2=0x7fb5480f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 --2- 172.21.15.46:0/2045606203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5480f6bf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fb538004660 tx=0x7fb538009b30).stop 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 -- 172.21.15.46:0/2045606203 shutdown_connections 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 --2- 172.21.15.46:0/2045606203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5480f7a70 0x7fb5480f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 --2- 172.21.15.46:0/2045606203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5480f6bf0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 -- 172.21.15.46:0/2045606203 >> 172.21.15.46:0/2045606203 conn(0x7fb5480f1f50 msgr2=0x7fb5480f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 -- 172.21.15.46:0/2045606203 shutdown_connections 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.641+0000 7fb54e586700 1 -- 172.21.15.46:0/2045606203 wait complete. 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb54e586700 1 Processor -- start 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb54e586700 1 -- start start 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb54e586700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5481070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb54e586700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5480f7a70 0x7fb5481075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb54e586700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb548107b50 con 0x7fb5480f67d0 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb54e586700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb548107c90 con 0x7fb5480f7a70 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb547fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5481070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb547fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5481070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40400/0 (socket says 172.21.15.46:40400) 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb547fff700 1 -- 172.21.15.46:0/3416137327 learned_addr learned my addr 172.21.15.46:0/3416137327 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.642+0000 7fb5477fe700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5480f7a70 0x7fb5481075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb547fff700 1 -- 172.21.15.46:0/3416137327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5480f7a70 msgr2=0x7fb5481075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb547fff700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5480f7a70 0x7fb5481075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb547fff700 1 -- 172.21.15.46:0/3416137327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538005040 con 0x7fb5480f67d0 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb547fff700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5481070b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fb538009f50 tx=0x7fb538009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb538018070 con 0x7fb5480f67d0 2024-01-21T08:18:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5481085d0 con 0x7fb5480f67d0 2024-01-21T08:18:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb538013410 con 0x7fb5480f67d0 2024-01-21T08:18:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb53801c3c0 con 0x7fb5480f67d0 2024-01-21T08:18:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.643+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb548108b30 con 0x7fb5480f67d0 2024-01-21T08:18:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.644+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb54805dd10 con 0x7fb5480f67d0 2024-01-21T08:18:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.644+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb53801d420 con 0x7fb5480f67d0 2024-01-21T08:18:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb544ff9700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb53005c2f0 0x7fb53005e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb538077990 con 0x7fb5480f67d0 2024-01-21T08:18:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb5477fe700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb53005c2f0 0x7fb53005e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb544ff9700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] conn(0x7fb530062080 0x7fb5300644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 --> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb530064b60 con 0x7fb530062080 2024-01-21T08:18:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb53807c200 con 0x7fb5480f67d0 2024-01-21T08:18:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb54cb23700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] conn(0x7fb530062080 0x7fb5300644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb5477fe700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb53005c2f0 0x7fb53005e7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb53c0041d0 tx=0x7fb53c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.645+0000 7fb54cb23700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] conn(0x7fb530062080 0x7fb5300644e0 crc :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.646+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== osd.0 v2:172.21.15.46:6802/2710852688 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fb530064b60 con 0x7fb530062080 2024-01-21T08:18:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.660+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 --> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb548108f90 con 0x7fb530062080 2024-01-21T08:18:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.660+0000 7fb544ff9700 1 -- 172.21.15.46:0/3416137327 <== osd.0 v2:172.21.15.46:6802/2710852688 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb548108f90 con 0x7fb530062080 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] conn(0x7fb530062080 msgr2=0x7fb5300644e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] conn(0x7fb530062080 0x7fb5300644e0 crc :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb53005c2f0 msgr2=0x7fb53005e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb53005c2f0 0x7fb53005e7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb53c0041d0 tx=0x7fb53c009040).stop 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 msgr2=0x7fb5481070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5481070b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fb538009f50 tx=0x7fb538009b30).stop 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 shutdown_connections 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6802/2710852688,v1:172.21.15.46:6803/2710852688] conn(0x7fb530062080 0x7fb5300644e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb53005c2f0 0x7fb53005e7b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5480f67d0 0x7fb5481070b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 --2- 172.21.15.46:0/3416137327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5480f7a70 0x7fb5481075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 >> 172.21.15.46:0/3416137327 conn(0x7fb5480f1f50 msgr2=0x7fb5480fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 shutdown_connections 2024-01-21T08:18:32.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:32.661+0000 7fb54e586700 1 -- 172.21.15.46:0/3416137327 wait complete. 2024-01-21T08:18:33.037 INFO:teuthology.orchestra.run.smithi046.stdout:38654705755 2024-01-21T08:18:33.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-01-21T08:18:33.265 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:33.699 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:32.096593+0000 mgr.smithi046.lmxnwc (mgr.14164) 443 : cluster [DBG] pgmap v422: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:33.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:32.096593+0000 mgr.smithi046.lmxnwc (mgr.14164) 443 : cluster [DBG] pgmap v422: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 -- 172.21.15.46:0/3179645727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f6980 msgr2=0x7f79e40f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/3179645727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f6980 0x7f79e40f6de0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f79d4004660 tx=0x7f79d4009b30).stop 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 -- 172.21.15.46:0/3179645727 shutdown_connections 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/3179645727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f6980 0x7f79e40f6de0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/3179645727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e40f56e0 0x7f79e40f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 -- 172.21.15.46:0/3179645727 >> 172.21.15.46:0/3179645727 conn(0x7f79e40f0e20 msgr2=0x7f79e40f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 -- 172.21.15.46:0/3179645727 shutdown_connections 2024-01-21T08:18:34.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.208+0000 7f79ebbb7700 1 -- 172.21.15.46:0/3179645727 wait complete. 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79ebbb7700 1 Processor -- start 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79ebbb7700 1 -- start start 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79ebbb7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f56e0 0x7f79e4105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79ebbb7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e40f6980 0x7f79e4106490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79ebbb7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e4106a40 con 0x7f79e40f56e0 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79ebbb7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e4106b80 con 0x7f79e40f6980 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9953700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f56e0 0x7f79e4105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9953700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f56e0 0x7f79e4105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40426/0 (socket says 172.21.15.46:40426) 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9953700 1 -- 172.21.15.46:0/1512337708 learned_addr learned my addr 172.21.15.46:0/1512337708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9152700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e40f6980 0x7f79e4106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9953700 1 -- 172.21.15.46:0/1512337708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e40f6980 msgr2=0x7f79e4106490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9953700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e40f6980 0x7f79e4106490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9953700 1 -- 172.21.15.46:0/1512337708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d4005040 con 0x7f79e40f56e0 2024-01-21T08:18:34.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79e9953700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f56e0 0x7f79e4105fa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f79e000c530 tx=0x7f79e0007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:34.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.209+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e000f650 con 0x7f79e40f56e0 2024-01-21T08:18:34.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.210+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e4107520 con 0x7f79e40f56e0 2024-01-21T08:18:34.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.210+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79e000fc60 con 0x7f79e40f56e0 2024-01-21T08:18:34.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.210+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e0012c60 con 0x7f79e40f56e0 2024-01-21T08:18:34.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.210+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e4107ae0 con 0x7f79e40f56e0 2024-01-21T08:18:34.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.210+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f79e405dcf0 con 0x7f79e40f56e0 2024-01-21T08:18:34.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f79e0019030 con 0x7f79e40f56e0 2024-01-21T08:18:34.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79da7fc700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f79d005c2a0 0x7f79d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:34.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f79e0075740 con 0x7f79e40f56e0 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79e9152700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f79d005c2a0 0x7f79d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79da7fc700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] conn(0x7f79d0062030 0x7f79d0064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 --> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79d0064b10 con 0x7f79d0062030 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79e0077db0 con 0x7f79e40f56e0 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.211+0000 7f79ea154700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] conn(0x7f79d0062030 0x7f79d0064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.212+0000 7f79e9152700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f79d005c2a0 0x7f79d005e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f79d4004630 tx=0x7f79d4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.212+0000 7f79ea154700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] conn(0x7f79d0062030 0x7f79d0064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.212+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== osd.1 v2:172.21.15.46:6810/3872661900 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f79d0064b10 con 0x7f79d0062030 2024-01-21T08:18:34.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 --> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f79e4108020 con 0x7f79d0062030 2024-01-21T08:18:34.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79da7fc700 1 -- 172.21.15.46:0/1512337708 <== osd.1 v2:172.21.15.46:6810/3872661900 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f79e4108020 con 0x7f79d0062030 2024-01-21T08:18:34.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] conn(0x7f79d0062030 msgr2=0x7f79d0064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:34.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] conn(0x7f79d0062030 0x7f79d0064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f79d005c2a0 msgr2=0x7f79d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:34.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f79d005c2a0 0x7f79d005e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f79d4004630 tx=0x7f79d4009b30).stop 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f56e0 msgr2=0x7f79e4105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f56e0 0x7f79e4105fa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f79e000c530 tx=0x7f79e0007e10).stop 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 shutdown_connections 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6810/3872661900,v1:172.21.15.46:6811/3872661900] conn(0x7f79d0062030 0x7f79d0064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f79d005c2a0 0x7f79d005e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e40f56e0 0x7f79e4105fa0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 --2- 172.21.15.46:0/1512337708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e40f6980 0x7f79e4106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.227+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 >> 172.21.15.46:0/1512337708 conn(0x7f79e40f0e20 msgr2=0x7f79e40fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.228+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 shutdown_connections 2024-01-21T08:18:34.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:34.228+0000 7f79ebbb7700 1 -- 172.21.15.46:0/1512337708 wait complete. 2024-01-21T08:18:34.603 INFO:teuthology.orchestra.run.smithi046.stdout:55834574937 2024-01-21T08:18:34.604 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-01-21T08:18:34.839 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:35.633 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:34.096972+0000 mgr.smithi046.lmxnwc (mgr.14164) 444 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:35.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:34.096972+0000 mgr.smithi046.lmxnwc (mgr.14164) 444 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:35.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 -- 172.21.15.46:0/1634530461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f7a60 msgr2=0x7fac100f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:35.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 --2- 172.21.15.46:0/1634530461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f7a60 0x7fac100f7ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fabf8004660 tx=0x7fabf8009b30).stop 2024-01-21T08:18:35.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 -- 172.21.15.46:0/1634530461 shutdown_connections 2024-01-21T08:18:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 --2- 172.21.15.46:0/1634530461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f7a60 0x7fac100f7ec0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 --2- 172.21.15.46:0/1634530461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f67c0 0x7fac100f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 -- 172.21.15.46:0/1634530461 >> 172.21.15.46:0/1634530461 conn(0x7fac100f1f20 msgr2=0x7fac100f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 -- 172.21.15.46:0/1634530461 shutdown_connections 2024-01-21T08:18:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 -- 172.21.15.46:0/1634530461 wait complete. 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 Processor -- start 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.805+0000 7fac15933700 1 -- start start 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac15933700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f67c0 0x7fac1006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac15933700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f7a60 0x7fac10069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac15933700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1006a380 con 0x7fac100f67c0 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac15933700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1006a4c0 con 0x7fac100f7a60 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f67c0 0x7fac1006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f67c0 0x7fac1006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40444/0 (socket says 172.21.15.46:40444) 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0effd700 1 -- 172.21.15.46:0/4006980872 learned_addr learned my addr 172.21.15.46:0/4006980872 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0e7fc700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f7a60 0x7fac10069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0effd700 1 -- 172.21.15.46:0/4006980872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f7a60 msgr2=0x7fac10069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0effd700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f7a60 0x7fac10069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0effd700 1 -- 172.21.15.46:0/4006980872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf8005040 con 0x7fac100f67c0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac0effd700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f67c0 0x7fac1006b9a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fac0000c530 tx=0x7fac00007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac0000f650 con 0x7fac100f67c0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac0000fc60 con 0x7fac100f67c0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1006a7a0 con 0x7fac100f67c0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac00012d20 con 0x7fac100f67c0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.806+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1010f0e0 con 0x7fac100f67c0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.807+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac1005dd30 con 0x7fac100f67c0 2024-01-21T08:18:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fac00019030 con 0x7fac100f67c0 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac077fe700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fabfc05c2a0 0x7fabfc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fac00075e40 con 0x7fac100f67c0 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac0e7fc700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fabfc05c2a0 0x7fabfc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac077fe700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] conn(0x7fabfc062030 0x7fabfc064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 --> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabfc064b10 con 0x7fabfc062030 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac00076240 con 0x7fac100f67c0 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac0f7fe700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] conn(0x7fabfc062030 0x7fabfc064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.808+0000 7fac0e7fc700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fabfc05c2a0 0x7fabfc05e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fabf8009d60 tx=0x7fabf8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.809+0000 7fac0f7fe700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] conn(0x7fabfc062030 0x7fabfc064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.809+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== osd.2 v2:172.21.15.46:6818/609261533 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fabfc064b10 con 0x7fabfc062030 2024-01-21T08:18:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 --> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac10058150 con 0x7fabfc062030 2024-01-21T08:18:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac077fe700 1 -- 172.21.15.46:0/4006980872 <== osd.2 v2:172.21.15.46:6818/609261533 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fac10058150 con 0x7fabfc062030 2024-01-21T08:18:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] conn(0x7fabfc062030 msgr2=0x7fabfc064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] conn(0x7fabfc062030 0x7fabfc064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fabfc05c2a0 msgr2=0x7fabfc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fabfc05c2a0 0x7fabfc05e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fabf8009d60 tx=0x7fabf8009b30).stop 2024-01-21T08:18:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f67c0 msgr2=0x7fac1006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f67c0 0x7fac1006b9a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fac0000c530 tx=0x7fac00007e10).stop 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 shutdown_connections 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6818/609261533,v1:172.21.15.46:6819/609261533] conn(0x7fabfc062030 0x7fabfc064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fabfc05c2a0 0x7fabfc05e760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac100f67c0 0x7fac1006b9a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 --2- 172.21.15.46:0/4006980872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac100f7a60 0x7fac10069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 >> 172.21.15.46:0/4006980872 conn(0x7fac100f1f20 msgr2=0x7fac100fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 shutdown_connections 2024-01-21T08:18:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:35.824+0000 7fac15933700 1 -- 172.21.15.46:0/4006980872 wait complete. 2024-01-21T08:18:36.203 INFO:teuthology.orchestra.run.smithi046.stdout:73014444118 2024-01-21T08:18:36.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-01-21T08:18:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2594615317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c0f9920 msgr2=0x7f4f8c0fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2594615317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c0f9920 0x7f4f8c0fbd70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f4f7c004660 tx=0x7f4f7c009b30).stop 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2594615317 shutdown_connections 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2594615317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c0f9920 0x7f4f8c0fbd70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2594615317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8c060870 0x7f4f8c0f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2594615317 >> 172.21.15.46:0/2594615317 conn(0x7f4f8c0f0e20 msgr2=0x7f4f8c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2594615317 shutdown_connections 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2594615317 wait complete. 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.433+0000 7f4f90e4f700 1 Processor -- start 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f90e4f700 1 -- start start 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f90e4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c060870 0x7f4f8c0f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f90e4f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8c0f9920 0x7f4f8c0f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f90e4f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c0f5800 con 0x7f4f8c060870 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f90e4f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c0f5940 con 0x7f4f8c0f9920 2024-01-21T08:18:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f8a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c060870 0x7f4f8c0f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f8a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c060870 0x7f4f8c0f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40452/0 (socket says 172.21.15.46:40452) 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f8a59c700 1 -- 172.21.15.46:0/2234073079 learned_addr learned my addr 172.21.15.46:0/2234073079 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f89d9b700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8c0f9920 0x7f4f8c0f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f8a59c700 1 -- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8c0f9920 msgr2=0x7f4f8c0f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f8a59c700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8c0f9920 0x7f4f8c0f52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f8a59c700 1 -- 172.21.15.46:0/2234073079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c005040 con 0x7f4f8c060870 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f8a59c700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c060870 0x7f4f8c0f6e20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f4f800026d0 tx=0x7f4f80009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f800079c0 con 0x7f4f8c060870 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f8000f040 con 0x7f4f8c060870 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c0f5c20 con 0x7f4f8c060870 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f800148b0 con 0x7f4f8c060870 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.434+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c10f130 con 0x7f4f8c060870 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.435+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f8c05dd30 con 0x7f4f8c060870 2024-01-21T08:18:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.436+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4f80007b20 con 0x7f4f8c060870 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.436+0000 7f4f7affd700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4f7405c2a0 0x7f4f7405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.436+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4f80075b00 con 0x7f4f8c060870 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.436+0000 7f4f89d9b700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4f7405c2a0 0x7f4f7405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.436+0000 7f4f7affd700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] conn(0x7f4f74062030 0x7f4f74064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.436+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 --> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f74064b10 con 0x7f4f74062030 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.436+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f8007a220 con 0x7f4f8c060870 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.437+0000 7f4f89d9b700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4f7405c2a0 0x7f4f7405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4f7c004e30 tx=0x7f4f7c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.437+0000 7f4f8ad9d700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] conn(0x7f4f74062030 0x7f4f74064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.437+0000 7f4f8ad9d700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] conn(0x7f4f74062030 0x7f4f74064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.438+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== osd.3 v2:172.21.15.78:6800/1326350774 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f4f74064b10 con 0x7f4f74062030 2024-01-21T08:18:37.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.452+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 --> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4f8c058150 con 0x7f4f74062030 2024-01-21T08:18:37.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.452+0000 7f4f7affd700 1 -- 172.21.15.46:0/2234073079 <== osd.3 v2:172.21.15.78:6800/1326350774 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4f8c058150 con 0x7f4f74062030 2024-01-21T08:18:37.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] conn(0x7f4f74062030 msgr2=0x7f4f74064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] conn(0x7f4f74062030 0x7f4f74064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4f7405c2a0 msgr2=0x7f4f7405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4f7405c2a0 0x7f4f7405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4f7c004e30 tx=0x7f4f7c009b30).stop 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c060870 msgr2=0x7f4f8c0f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c060870 0x7f4f8c0f6e20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f4f800026d0 tx=0x7f4f80009de0).stop 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 shutdown_connections 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:6800/1326350774,v1:172.21.15.78:6801/1326350774] conn(0x7f4f74062030 0x7f4f74064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4f7405c2a0 0x7f4f7405e760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f8c060870 0x7f4f8c0f6e20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 --2- 172.21.15.46:0/2234073079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8c0f9920 0x7f4f8c0f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 >> 172.21.15.46:0/2234073079 conn(0x7f4f8c0f0e20 msgr2=0x7f4f8c0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 shutdown_connections 2024-01-21T08:18:37.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:37.453+0000 7f4f90e4f700 1 -- 172.21.15.46:0/2234073079 wait complete. 2024-01-21T08:18:37.475 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:36.097294+0000 mgr.smithi046.lmxnwc (mgr.14164) 445 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:36.097294+0000 mgr.smithi046.lmxnwc (mgr.14164) 445 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:37.778 INFO:teuthology.orchestra.run.smithi046.stdout:107374182441 2024-01-21T08:18:37.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-01-21T08:18:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 -- 172.21.15.46:0/2177329861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f6960 msgr2=0x7f98080f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 --2- 172.21.15.46:0/2177329861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f6960 0x7f98080f6dc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f97fc004660 tx=0x7f97fc009b30).stop 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 -- 172.21.15.46:0/2177329861 shutdown_connections 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 --2- 172.21.15.46:0/2177329861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f6960 0x7f98080f6dc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 --2- 172.21.15.46:0/2177329861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98080f56c0 0x7f98080f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 -- 172.21.15.46:0/2177329861 >> 172.21.15.46:0/2177329861 conn(0x7f98080f0e20 msgr2=0x7f98080f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 -- 172.21.15.46:0/2177329861 shutdown_connections 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.048+0000 7f980f71d700 1 -- 172.21.15.46:0/2177329861 wait complete. 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980f71d700 1 Processor -- start 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980f71d700 1 -- start start 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980f71d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f56c0 0x7f9808109d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980f71d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98080f6960 0x7f980810c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980f71d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9808060920 con 0x7f98080f56c0 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980f71d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9808060a60 con 0x7f98080f6960 2024-01-21T08:18:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980d4b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f56c0 0x7f9808109d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980d4b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f56c0 0x7f9808109d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40470/0 (socket says 172.21.15.46:40470) 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980d4b9700 1 -- 172.21.15.46:0/3510193598 learned_addr learned my addr 172.21.15.46:0/3510193598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.049+0000 7f980ccb8700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98080f6960 0x7f980810c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f980d4b9700 1 -- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98080f6960 msgr2=0x7f980810c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f980d4b9700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98080f6960 0x7f980810c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f980d4b9700 1 -- 172.21.15.46:0/3510193598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc005040 con 0x7f98080f56c0 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f980d4b9700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f56c0 0x7f9808109d40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f980400e7c0 tx=0x7f980400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98040092e0 con 0x7f98080f56c0 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f980400f040 con 0x7f98080f56c0 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980810c7f0 con 0x7f98080f56c0 2024-01-21T08:18:39.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9804014750 con 0x7f98080f56c0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.050+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980810cd80 con 0x7f98080f56c0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.051+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f980805dd30 con 0x7f98080f56c0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.051+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f980401b030 con 0x7f98080f56c0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f97f9ffb700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f97f405c2a0 0x7f97f405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9804075b30 con 0x7f98080f56c0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f980ccb8700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f97f405c2a0 0x7f97f405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f97f9ffb700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] conn(0x7f97f4062030 0x7f97f4064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 --> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97f4064b10 con 0x7f97f4062030 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f980407a1c0 con 0x7f98080f56c0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f980ccb8700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f97f405c2a0 0x7f97f405e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f97fc004d80 tx=0x7f97fc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.052+0000 7f980dcba700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] conn(0x7f97f4062030 0x7f97f4064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.053+0000 7f980dcba700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] conn(0x7f97f4062030 0x7f97f4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:39.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.053+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== osd.4 v2:172.21.15.78:6808/818610611 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f97f4064b10 con 0x7f97f4062030 2024-01-21T08:18:39.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 --> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f98080581d0 con 0x7f97f4062030 2024-01-21T08:18:39.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f97f9ffb700 1 -- 172.21.15.46:0/3510193598 <== osd.4 v2:172.21.15.78:6808/818610611 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f98080581d0 con 0x7f97f4062030 2024-01-21T08:18:39.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] conn(0x7f97f4062030 msgr2=0x7f97f4064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:39.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f980f71d700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] conn(0x7f97f4062030 0x7f97f4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f97f405c2a0 msgr2=0x7f97f405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f980f71d700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f97f405c2a0 0x7f97f405e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f97fc004d80 tx=0x7f97fc00b040).stop 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f56c0 msgr2=0x7f9808109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.068+0000 7f980f71d700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f56c0 0x7f9808109d40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f980400e7c0 tx=0x7f980400b040).stop 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 shutdown_connections 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:6808/818610611,v1:172.21.15.78:6809/818610611] conn(0x7f97f4062030 0x7f97f4064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f97f405c2a0 0x7f97f405e760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98080f56c0 0x7f9808109d40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 --2- 172.21.15.46:0/3510193598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98080f6960 0x7f980810c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 >> 172.21.15.46:0/3510193598 conn(0x7f98080f0e20 msgr2=0x7f98080fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 shutdown_connections 2024-01-21T08:18:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:39.069+0000 7f980f71d700 1 -- 172.21.15.46:0/3510193598 wait complete. 2024-01-21T08:18:39.446 INFO:teuthology.orchestra.run.smithi046.stdout:133143986186 2024-01-21T08:18:39.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-01-21T08:18:39.675 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:39.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:38.097758+0000 mgr.smithi046.lmxnwc (mgr.14164) 446 : cluster [DBG] pgmap v425: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:39.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:38.097758+0000 mgr.smithi046.lmxnwc (mgr.14164) 446 : cluster [DBG] pgmap v425: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 -- 172.21.15.46:0/2153220571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 msgr2=0x7f78240f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 --2- 172.21.15.46:0/2153220571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f6c10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f7814004660 tx=0x7f7814009b30).stop 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 -- 172.21.15.46:0/2153220571 shutdown_connections 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 --2- 172.21.15.46:0/2153220571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f6c10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 --2- 172.21.15.46:0/2153220571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240f5510 0x7f78240f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 -- 172.21.15.46:0/2153220571 >> 172.21.15.46:0/2153220571 conn(0x7f78240f0c70 msgr2=0x7f78240f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 -- 172.21.15.46:0/2153220571 shutdown_connections 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 -- 172.21.15.46:0/2153220571 wait complete. 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 Processor -- start 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.683+0000 7f782b41e700 1 -- start start 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782b41e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240f5510 0x7f78240f9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782b41e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782b41e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240f8050 con 0x7f78240f67b0 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782b41e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240f8190 con 0x7f78240f5510 2024-01-21T08:18:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f7829c1b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f7829c1b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35118/0 (socket says 172.21.15.46:35118) 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f7829c1b700 1 -- 172.21.15.46:0/1661715312 learned_addr learned my addr 172.21.15.46:0/1661715312 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782a41c700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240f5510 0x7f78240f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f7829c1b700 1 -- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240f5510 msgr2=0x7f78240f9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f7829c1b700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240f5510 0x7f78240f9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f7829c1b700 1 -- 172.21.15.46:0/1661715312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7814005040 con 0x7f78240f67b0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782a41c700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240f5510 0x7f78240f9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f7829c1b700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f7af0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f7814009830 tx=0x7f7814009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7814018070 con 0x7f78240f67b0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7814013410 con 0x7f78240f67b0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78240f8410 con 0x7f78240f67b0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f781401c620 con 0x7f78240f67b0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.684+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78240f8970 con 0x7f78240f67b0 2024-01-21T08:18:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.685+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f782405db80 con 0x7f78240f67b0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f781401c890 con 0x7f78240f67b0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f781affd700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f781005c2b0 0x7f781005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f78140779b0 con 0x7f78240f67b0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f782a41c700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f781005c2b0 0x7f781005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f781affd700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] conn(0x7f7810062040 0x7f78100644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 --> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7810064b20 con 0x7f7810062040 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f781407c220 con 0x7f78240f67b0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.686+0000 7f782a41c700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f781005c2b0 0x7f781005e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f78240f9400 tx=0x7f7820009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.687+0000 7f782ac1d700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] conn(0x7f7810062040 0x7f78100644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.687+0000 7f782ac1d700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] conn(0x7f7810062040 0x7f78100644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.688+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== osd.5 v2:172.21.15.78:6816/1959692294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f7810064b20 con 0x7f7810062040 2024-01-21T08:18:40.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.702+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 --> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f78240faeb0 con 0x7f7810062040 2024-01-21T08:18:40.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f781affd700 1 -- 172.21.15.46:0/1661715312 <== osd.5 v2:172.21.15.78:6816/1959692294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f78240faeb0 con 0x7f7810062040 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] conn(0x7f7810062040 msgr2=0x7f78100644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] conn(0x7f7810062040 0x7f78100644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f781005c2b0 msgr2=0x7f781005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f781005c2b0 0x7f781005e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f78240f9400 tx=0x7f7820009040).stop 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 msgr2=0x7f78240f7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f7af0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f7814009830 tx=0x7f7814009b30).stop 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 shutdown_connections 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:6816/1959692294,v1:172.21.15.78:6817/1959692294] conn(0x7f7810062040 0x7f78100644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f781005c2b0 0x7f781005e770 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240f5510 0x7f78240f9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 --2- 172.21.15.46:0/1661715312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78240f67b0 0x7f78240f7af0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:40.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 >> 172.21.15.46:0/1661715312 conn(0x7f78240f0c70 msgr2=0x7f7824060380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:40.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 shutdown_connections 2024-01-21T08:18:40.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:40.703+0000 7f782b41e700 1 -- 172.21.15.46:0/1661715312 wait complete. 2024-01-21T08:18:41.077 INFO:teuthology.orchestra.run.smithi046.stdout:150323855368 2024-01-21T08:18:41.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-01-21T08:18:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:40.098267+0000 mgr.smithi046.lmxnwc (mgr.14164) 447 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:40.098267+0000 mgr.smithi046.lmxnwc (mgr.14164) 447 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/3843811197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80607f0 msgr2=0x7f9eb80fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/3843811197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80607f0 0x7f9eb80fb820 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f9ea8004690 tx=0x7f9ea8009b30).stop 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/3843811197 shutdown_connections 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/3843811197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eb80fbd10 0x7f9eb80fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/3843811197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80607f0 0x7f9eb80fb820 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/3843811197 >> 172.21.15.46:0/3843811197 conn(0x7f9eb80f0e20 msgr2=0x7f9eb80f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/3843811197 shutdown_connections 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/3843811197 wait complete. 2024-01-21T08:18:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 Processor -- start 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.294+0000 7f9ebf8b7700 1 -- start start 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebf8b7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eb80607f0 0x7f9eb81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebf8b7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80fbd10 0x7f9eb81097e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebf8b7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8109e20 con 0x7f9eb80fbd10 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebf8b7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb810a6d0 con 0x7f9eb80607f0 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebce52700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80fbd10 0x7f9eb81097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebce52700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80fbd10 0x7f9eb81097e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35144/0 (socket says 172.21.15.46:35144) 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebce52700 1 -- 172.21.15.46:0/4029487180 learned_addr learned my addr 172.21.15.46:0/4029487180 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebd653700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eb80607f0 0x7f9eb81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebce52700 1 -- 172.21.15.46:0/4029487180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eb80607f0 msgr2=0x7f9eb81092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebce52700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eb80607f0 0x7f9eb81092f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebce52700 1 -- 172.21.15.46:0/4029487180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8005040 con 0x7f9eb80fbd10 2024-01-21T08:18:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebce52700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80fbd10 0x7f9eb81097e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f9eb40093f0 tx=0x7f9eb4006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9eadffb700 1 -- 172.21.15.46:0/4029487180 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb4013640 con 0x7f9eb80fbd10 2024-01-21T08:18:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9eadffb700 1 -- 172.21.15.46:0/4029487180 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9eb4013c50 con 0x7f9eb80fbd10 2024-01-21T08:18:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb810a9b0 con 0x7f9eb80fbd10 2024-01-21T08:18:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.295+0000 7f9eadffb700 1 -- 172.21.15.46:0/4029487180 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb4012c80 con 0x7f9eb80fbd10 2024-01-21T08:18:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.296+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb810aef0 con 0x7f9eb80fbd10 2024-01-21T08:18:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.296+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb8059100 con 0x7f9eb80fbd10 2024-01-21T08:18:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.297+0000 7f9eadffb700 1 -- 172.21.15.46:0/4029487180 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9eb401a060 con 0x7f9eb80fbd10 2024-01-21T08:18:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.297+0000 7f9eadffb700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9ea405c2a0 0x7f9ea405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.297+0000 7f9eadffb700 1 -- 172.21.15.46:0/4029487180 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9eb4075ad0 con 0x7f9eb80fbd10 2024-01-21T08:18:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.297+0000 7f9ebd653700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9ea405c2a0 0x7f9ea405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.297+0000 7f9ebd653700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9ea405c2a0 0x7f9ea405e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9ea8005010 tx=0x7f9ea8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.301+0000 7f9eadffb700 1 -- 172.21.15.46:0/4029487180 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9eb4048af0 con 0x7f9eb80fbd10 2024-01-21T08:18:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.460+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9eb8101200 con 0x7f9eb80fbd10 2024-01-21T08:18:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.460+0000 7f9eadffb700 1 -- 172.21.15.46:0/4029487180 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9eb404c110 con 0x7f9eb80fbd10 2024-01-21T08:18:42.463 INFO:teuthology.orchestra.run.smithi046.stdout:38654705757 2024-01-21T08:18:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9ea405c2a0 msgr2=0x7f9ea405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9ea405c2a0 0x7f9ea405e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9ea8005010 tx=0x7f9ea8009b30).stop 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80fbd10 msgr2=0x7f9eb81097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80fbd10 0x7f9eb81097e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f9eb40093f0 tx=0x7f9eb4006ec0).stop 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 shutdown_connections 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9ea405c2a0 0x7f9ea405e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eb80607f0 0x7f9eb81092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 --2- 172.21.15.46:0/4029487180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9eb80fbd10 0x7f9eb81097e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 >> 172.21.15.46:0/4029487180 conn(0x7f9eb80f0e20 msgr2=0x7f9eb80f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 shutdown_connections 2024-01-21T08:18:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:42.462+0000 7f9ebf8b7700 1 -- 172.21.15.46:0/4029487180 wait complete. 2024-01-21T08:18:42.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:42.098559+0000 mgr.smithi046.lmxnwc (mgr.14164) 448 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:42.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:42.098559+0000 mgr.smithi046.lmxnwc (mgr.14164) 448 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:42.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705755 got 38654705757 for osd.0 2024-01-21T08:18:42.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-01-21T08:18:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:43.686 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:42.461960+0000 mon.smithi046 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.46:0/4029487180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-21T08:18:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:42.461960+0000 mon.smithi046 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.46:0/4029487180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.069+0000 7f0a9759e700 1 -- 172.21.15.46:0/2917922061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 msgr2=0x7f0a980f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.069+0000 7f0a9759e700 1 --2- 172.21.15.46:0/2917922061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a980f5930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f0a88004660 tx=0x7f0a88009b30).stop 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 -- 172.21.15.46:0/2917922061 shutdown_connections 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 --2- 172.21.15.46:0/2917922061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a980f67b0 0x7f0a980f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 --2- 172.21.15.46:0/2917922061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a980f5930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 -- 172.21.15.46:0/2917922061 >> 172.21.15.46:0/2917922061 conn(0x7f0a980f0c70 msgr2=0x7f0a980f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 -- 172.21.15.46:0/2917922061 shutdown_connections 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 -- 172.21.15.46:0/2917922061 wait complete. 2024-01-21T08:18:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 Processor -- start 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 -- start start 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a98106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a980f67b0 0x7f0a981073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98107980 con 0x7f0a980f5510 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.070+0000 7f0a9759e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98107ac0 con 0x7f0a980f67b0 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a98106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a98106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35166/0 (socket says 172.21.15.46:35166) 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9659c700 1 -- 172.21.15.46:0/1091498223 learned_addr learned my addr 172.21.15.46:0/1091498223 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a95d9b700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a980f67b0 0x7f0a981073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9659c700 1 -- 172.21.15.46:0/1091498223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a980f67b0 msgr2=0x7f0a981073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9659c700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a980f67b0 0x7f0a981073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9659c700 1 -- 172.21.15.46:0/1091498223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a88005040 con 0x7f0a980f5510 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9659c700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a98106ee0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f0a88004e30 tx=0x7f0a88009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a86ffd700 1 -- 172.21.15.46:0/1091498223 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a88018070 con 0x7f0a980f5510 2024-01-21T08:18:44.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a98108400 con 0x7f0a980f5510 2024-01-21T08:18:44.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a86ffd700 1 -- 172.21.15.46:0/1091498223 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a88013410 con 0x7f0a980f5510 2024-01-21T08:18:44.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a86ffd700 1 -- 172.21.15.46:0/1091498223 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a8801de20 con 0x7f0a980f5510 2024-01-21T08:18:44.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.071+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a98108960 con 0x7f0a980f5510 2024-01-21T08:18:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.072+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a98045ad0 con 0x7f0a980f5510 2024-01-21T08:18:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.073+0000 7f0a86ffd700 1 -- 172.21.15.46:0/1091498223 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0a8801d420 con 0x7f0a980f5510 2024-01-21T08:18:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.073+0000 7f0a86ffd700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0a8005c2b0 0x7f0a8005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.073+0000 7f0a86ffd700 1 -- 172.21.15.46:0/1091498223 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0a880777b0 con 0x7f0a980f5510 2024-01-21T08:18:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.073+0000 7f0a95d9b700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0a8005c2b0 0x7f0a8005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.073+0000 7f0a95d9b700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0a8005c2b0 0x7f0a8005e770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0a8c004c40 tx=0x7f0a8c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.077+0000 7f0a86ffd700 1 -- 172.21.15.46:0/1091498223 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a8804a7b0 con 0x7f0a980f5510 2024-01-21T08:18:44.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.235+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0a98109120 con 0x7f0a980f5510 2024-01-21T08:18:44.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.236+0000 7f0a86ffd700 1 -- 172.21.15.46:0/1091498223 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a8804ddd0 con 0x7f0a980f5510 2024-01-21T08:18:44.237 INFO:teuthology.orchestra.run.smithi046.stdout:55834574938 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0a8005c2b0 msgr2=0x7f0a8005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0a8005c2b0 0x7f0a8005e770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0a8c004c40 tx=0x7f0a8c00b040).stop 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 msgr2=0x7f0a98106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a98106ee0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f0a88004e30 tx=0x7f0a88009b30).stop 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 shutdown_connections 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f0a8005c2b0 0x7f0a8005e770 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980f5510 0x7f0a98106ee0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 --2- 172.21.15.46:0/1091498223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a980f67b0 0x7f0a981073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 >> 172.21.15.46:0/1091498223 conn(0x7f0a980f0c70 msgr2=0x7f0a980603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:44.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 shutdown_connections 2024-01-21T08:18:44.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:44.237+0000 7f0a9759e700 1 -- 172.21.15.46:0/1091498223 wait complete. 2024-01-21T08:18:44.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:44.098857+0000 mgr.smithi046.lmxnwc (mgr.14164) 449 : cluster [DBG] pgmap v428: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:44.534 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:44.236570+0000 mon.smithi046 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.46:0/1091498223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-21T08:18:44.619 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574937 got 55834574938 for osd.1 2024-01-21T08:18:44.619 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-01-21T08:18:44.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:44 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:44.098857+0000 mgr.smithi046.lmxnwc (mgr.14164) 449 : cluster [DBG] pgmap v428: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:44.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:44 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:44.236570+0000 mon.smithi046 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.46:0/1091498223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-21T08:18:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 -- 172.21.15.46:0/1727927474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f7a70 msgr2=0x7f611c0f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 --2- 172.21.15.46:0/1727927474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f7a70 0x7f611c0f7ed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f6110004660 tx=0x7f6110009b30).stop 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 -- 172.21.15.46:0/1727927474 shutdown_connections 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 --2- 172.21.15.46:0/1727927474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f7a70 0x7f611c0f7ed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 --2- 172.21.15.46:0/1727927474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f67d0 0x7f611c0f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 -- 172.21.15.46:0/1727927474 >> 172.21.15.46:0/1727927474 conn(0x7f611c0f1f50 msgr2=0x7f611c0f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 -- 172.21.15.46:0/1727927474 shutdown_connections 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.867+0000 7f6120d66700 1 -- 172.21.15.46:0/1727927474 wait complete. 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6120d66700 1 Processor -- start 2024-01-21T08:18:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6120d66700 1 -- start start 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6120d66700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f67d0 0x7f611c107000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6120d66700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f7a70 0x7f611c1074f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6120d66700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c107aa0 con 0x7f611c0f7a70 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6120d66700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c107be0 con 0x7f611c0f67d0 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6119d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f7a70 0x7f611c1074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6119d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f7a70 0x7f611c1074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35178/0 (socket says 172.21.15.46:35178) 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f611a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f67d0 0x7f611c107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6119d9b700 1 -- 172.21.15.46:0/2550755244 learned_addr learned my addr 172.21.15.46:0/2550755244 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6119d9b700 1 -- 172.21.15.46:0/2550755244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f67d0 msgr2=0x7f611c107000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6119d9b700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f67d0 0x7f611c107000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6119d9b700 1 -- 172.21.15.46:0/2550755244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6110005040 con 0x7f611c0f7a70 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f6119d9b700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f7a70 0x7f611c1074f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f6110000c00 tx=0x7f611000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.868+0000 7f610affd700 1 -- 172.21.15.46:0/2550755244 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6110019070 con 0x7f611c0f7a70 2024-01-21T08:18:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.869+0000 7f610affd700 1 -- 172.21.15.46:0/2550755244 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6110007e70 con 0x7f611c0f7a70 2024-01-21T08:18:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.869+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c108520 con 0x7f611c0f7a70 2024-01-21T08:18:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.869+0000 7f610affd700 1 -- 172.21.15.46:0/2550755244 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6110014c90 con 0x7f611c0f7a70 2024-01-21T08:18:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.869+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c108a80 con 0x7f611c0f7a70 2024-01-21T08:18:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.869+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c045ad0 con 0x7f611c0f7a70 2024-01-21T08:18:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.870+0000 7f610affd700 1 -- 172.21.15.46:0/2550755244 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6110025070 con 0x7f611c0f7a70 2024-01-21T08:18:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.870+0000 7f610affd700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f610405c2a0 0x7f610405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.870+0000 7f610affd700 1 -- 172.21.15.46:0/2550755244 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6110076be0 con 0x7f611c0f7a70 2024-01-21T08:18:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.870+0000 7f611a59c700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f610405c2a0 0x7f610405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.871+0000 7f611a59c700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f610405c2a0 0x7f610405e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f610c009fd0 tx=0x7f610c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:45.874+0000 7f610affd700 1 -- 172.21.15.46:0/2550755244 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6110049be0 con 0x7f611c0f7a70 2024-01-21T08:18:46.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.034+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f611c027a00 con 0x7f611c0f7a70 2024-01-21T08:18:46.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.034+0000 7f610affd700 1 -- 172.21.15.46:0/2550755244 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f611004d200 con 0x7f611c0f7a70 2024-01-21T08:18:46.036 INFO:teuthology.orchestra.run.smithi046.stdout:73014444120 2024-01-21T08:18:46.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f610405c2a0 msgr2=0x7f610405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f610405c2a0 0x7f610405e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f610c009fd0 tx=0x7f610c00b040).stop 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f7a70 msgr2=0x7f611c1074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f7a70 0x7f611c1074f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f6110000c00 tx=0x7f611000b040).stop 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 shutdown_connections 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f610405c2a0 0x7f610405e760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f611c0f67d0 0x7f611c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 --2- 172.21.15.46:0/2550755244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f611c0f7a70 0x7f611c1074f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 >> 172.21.15.46:0/2550755244 conn(0x7f611c0f1f50 msgr2=0x7f611c0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 shutdown_connections 2024-01-21T08:18:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:46.036+0000 7f6120d66700 1 -- 172.21.15.46:0/2550755244 wait complete. 2024-01-21T08:18:46.334 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:46 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:46.035273+0000 mon.smithi046 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.46:0/2550755244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-21T08:18:46.411 INFO:tasks.cephadm.ceph_manager.ceph:need seq 73014444118 got 73014444120 for osd.2 2024-01-21T08:18:46.411 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-01-21T08:18:46.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:46 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:46.035273+0000 mon.smithi046 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.46:0/2550755244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-21T08:18:46.640 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:47.456 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:46.099140+0000 mgr.smithi046.lmxnwc (mgr.14164) 450 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:47.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:46.099140+0000 mgr.smithi046.lmxnwc (mgr.14164) 450 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:47.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 -- 172.21.15.46:0/352113316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 msgr2=0x7feb280f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:47.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 --2- 172.21.15.46:0/352113316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb280f7ee0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7feb18004660 tx=0x7feb18009b30).stop 2024-01-21T08:18:47.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 -- 172.21.15.46:0/352113316 shutdown_connections 2024-01-21T08:18:47.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 --2- 172.21.15.46:0/352113316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb280f7ee0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:47.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 --2- 172.21.15.46:0/352113316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280f67e0 0x7feb280f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:47.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 -- 172.21.15.46:0/352113316 >> 172.21.15.46:0/352113316 conn(0x7feb280f1f80 msgr2=0x7feb280f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 -- 172.21.15.46:0/352113316 shutdown_connections 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.645+0000 7feb2cca2700 1 -- 172.21.15.46:0/352113316 wait complete. 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb2cca2700 1 Processor -- start 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb2cca2700 1 -- start start 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb2cca2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280f67e0 0x7feb2806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb2cca2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb28069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb2cca2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2806a380 con 0x7feb280f7a80 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb2cca2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2806a4c0 con 0x7feb280f67e0 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb25d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb28069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb25d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb28069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35206/0 (socket says 172.21.15.46:35206) 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb25d9b700 1 -- 172.21.15.46:0/2306678566 learned_addr learned my addr 172.21.15.46:0/2306678566 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb2659c700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280f67e0 0x7feb2806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb25d9b700 1 -- 172.21.15.46:0/2306678566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280f67e0 msgr2=0x7feb2806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:47.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb25d9b700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280f67e0 0x7feb2806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:47.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb25d9b700 1 -- 172.21.15.46:0/2306678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18005040 con 0x7feb280f7a80 2024-01-21T08:18:47.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb25d9b700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb28069e20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7feb18009830 tx=0x7feb18009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:47.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.646+0000 7feb1effd700 1 -- 172.21.15.46:0/2306678566 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb18018070 con 0x7feb280f7a80 2024-01-21T08:18:47.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.647+0000 7feb1effd700 1 -- 172.21.15.46:0/2306678566 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb18013410 con 0x7feb280f7a80 2024-01-21T08:18:47.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.647+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2806a740 con 0x7feb280f7a80 2024-01-21T08:18:47.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.647+0000 7feb1effd700 1 -- 172.21.15.46:0/2306678566 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb1801c400 con 0x7feb280f7a80 2024-01-21T08:18:47.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.647+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2806aca0 con 0x7feb280f7a80 2024-01-21T08:18:47.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.647+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb28045ad0 con 0x7feb280f7a80 2024-01-21T08:18:47.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.648+0000 7feb1effd700 1 -- 172.21.15.46:0/2306678566 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7feb1801d420 con 0x7feb280f7a80 2024-01-21T08:18:47.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.648+0000 7feb1effd700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7feb1405c2a0 0x7feb1405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:47.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.648+0000 7feb1effd700 1 -- 172.21.15.46:0/2306678566 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7feb18077960 con 0x7feb280f7a80 2024-01-21T08:18:47.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.648+0000 7feb2659c700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7feb1405c2a0 0x7feb1405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:47.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.649+0000 7feb2659c700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7feb1405c2a0 0x7feb1405e760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7feb2806b730 tx=0x7feb10009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:47.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.652+0000 7feb1effd700 1 -- 172.21.15.46:0/2306678566 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb1804a9e0 con 0x7feb280f7a80 2024-01-21T08:18:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.810+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7feb28027a00 con 0x7feb280f7a80 2024-01-21T08:18:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.810+0000 7feb1effd700 1 -- 172.21.15.46:0/2306678566 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7feb1804e000 con 0x7feb280f7a80 2024-01-21T08:18:47.812 INFO:teuthology.orchestra.run.smithi046.stdout:107374182442 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7feb1405c2a0 msgr2=0x7feb1405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7feb1405c2a0 0x7feb1405e760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7feb2806b730 tx=0x7feb10009040).stop 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 msgr2=0x7feb28069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb28069e20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7feb18009830 tx=0x7feb18009b30).stop 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 shutdown_connections 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7feb1405c2a0 0x7feb1405e760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280f67e0 0x7feb2806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 --2- 172.21.15.46:0/2306678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280f7a80 0x7feb28069e20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 >> 172.21.15.46:0/2306678566 conn(0x7feb280f1f80 msgr2=0x7feb280fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.812+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 shutdown_connections 2024-01-21T08:18:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:47.813+0000 7feb2cca2700 1 -- 172.21.15.46:0/2306678566 wait complete. 2024-01-21T08:18:48.202 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182441 got 107374182442 for osd.3 2024-01-21T08:18:48.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-01-21T08:18:48.432 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:48.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:47.811469+0000 mon.smithi046 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.46:0/2306678566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-21T08:18:48.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:48 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:47.811469+0000 mon.smithi046 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.46:0/2306678566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-21T08:18:49.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:48.099663+0000 mgr.smithi046.lmxnwc (mgr.14164) 451 : cluster [DBG] pgmap v430: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/1045695961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a28060870 msgr2=0x7f9a280f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/1045695961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a28060870 0x7f9a280f93a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f9a1c004660 tx=0x7f9a1c009b30).stop 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/1045695961 shutdown_connections 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/1045695961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a280f9920 0x7f9a280fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/1045695961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a28060870 0x7f9a280f93a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/1045695961 >> 172.21.15.46:0/1045695961 conn(0x7f9a280f0e20 msgr2=0x7f9a280f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/1045695961 shutdown_connections 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.438+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/1045695961 wait complete. 2024-01-21T08:18:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a2d9ec700 1 Processor -- start 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a2d9ec700 1 -- start start 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a2d9ec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28060870 0x7f9a281070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a2d9ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a280f9920 0x7f9a281075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a2d9ec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28107b50 con 0x7f9a280f9920 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a2d9ec700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28107c90 con 0x7f9a28060870 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a267fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a280f9920 0x7f9a281075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a267fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a280f9920 0x7f9a281075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35220/0 (socket says 172.21.15.46:35220) 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a267fc700 1 -- 172.21.15.46:0/2401810184 learned_addr learned my addr 172.21.15.46:0/2401810184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a26ffd700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28060870 0x7f9a281070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a267fc700 1 -- 172.21.15.46:0/2401810184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28060870 msgr2=0x7f9a281070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a267fc700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28060870 0x7f9a281070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a267fc700 1 -- 172.21.15.46:0/2401810184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c005040 con 0x7f9a280f9920 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a267fc700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a280f9920 0x7f9a281075a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f9a1800c510 tx=0x7f9a18007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.439+0000 7f9a0f7fe700 1 -- 172.21.15.46:0/2401810184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a1800f650 con 0x7f9a280f9920 2024-01-21T08:18:49.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.440+0000 7f9a0f7fe700 1 -- 172.21.15.46:0/2401810184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a1800fc60 con 0x7f9a280f9920 2024-01-21T08:18:49.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.440+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a28108630 con 0x7f9a280f9920 2024-01-21T08:18:49.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.440+0000 7f9a0f7fe700 1 -- 172.21.15.46:0/2401810184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a18012c80 con 0x7f9a280f9920 2024-01-21T08:18:49.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.440+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a28108bf0 con 0x7f9a280f9920 2024-01-21T08:18:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.440+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a28045ad0 con 0x7f9a280f9920 2024-01-21T08:18:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.441+0000 7f9a0f7fe700 1 -- 172.21.15.46:0/2401810184 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9a18010070 con 0x7f9a280f9920 2024-01-21T08:18:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.441+0000 7f9a0f7fe700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a1005c2a0 0x7f9a1005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.441+0000 7f9a0f7fe700 1 -- 172.21.15.46:0/2401810184 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9a180757b0 con 0x7f9a280f9920 2024-01-21T08:18:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.441+0000 7f9a26ffd700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a1005c2a0 0x7f9a1005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.442+0000 7f9a26ffd700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a1005c2a0 0x7f9a1005e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9a1c013580 tx=0x7f9a1c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:49.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.445+0000 7f9a0f7fe700 1 -- 172.21.15.46:0/2401810184 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a180487d0 con 0x7f9a280f9920 2024-01-21T08:18:49.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:48.099663+0000 mgr.smithi046.lmxnwc (mgr.14164) 451 : cluster [DBG] pgmap v430: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:49.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.604+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9a28027a00 con 0x7f9a280f9920 2024-01-21T08:18:49.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.604+0000 7f9a0f7fe700 1 -- 172.21.15.46:0/2401810184 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9a1804bdf0 con 0x7f9a280f9920 2024-01-21T08:18:49.606 INFO:teuthology.orchestra.run.smithi046.stdout:133143986187 2024-01-21T08:18:49.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a1005c2a0 msgr2=0x7f9a1005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:49.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a1005c2a0 0x7f9a1005e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9a1c013580 tx=0x7f9a1c009b30).stop 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a280f9920 msgr2=0x7f9a281075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a280f9920 0x7f9a281075a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f9a1800c510 tx=0x7f9a18007df0).stop 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 shutdown_connections 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f9a1005c2a0 0x7f9a1005e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28060870 0x7f9a281070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 --2- 172.21.15.46:0/2401810184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a280f9920 0x7f9a281075a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 >> 172.21.15.46:0/2401810184 conn(0x7f9a280f0e20 msgr2=0x7f9a280f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 shutdown_connections 2024-01-21T08:18:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:49.606+0000 7f9a2d9ec700 1 -- 172.21.15.46:0/2401810184 wait complete. 2024-01-21T08:18:49.976 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986186 got 133143986187 for osd.4 2024-01-21T08:18:49.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-01-21T08:18:50.208 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:50.713 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:49.605008+0000 mon.smithi046 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.46:0/2401810184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-21T08:18:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:49.605008+0000 mon.smithi046 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.46:0/2401810184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 -- 172.21.15.46:0/2068463254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab300f9930 msgr2=0x7fab300fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 --2- 172.21.15.46:0/2068463254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab300f9930 0x7fab300fbd80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fab180045a0 tx=0x7fab18009b30).stop 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 -- 172.21.15.46:0/2068463254 shutdown_connections 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 --2- 172.21.15.46:0/2068463254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab300f9930 0x7fab300fbd80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 --2- 172.21.15.46:0/2068463254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab30060870 0x7fab300f93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 -- 172.21.15.46:0/2068463254 >> 172.21.15.46:0/2068463254 conn(0x7fab300f0e50 msgr2=0x7fab300f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 -- 172.21.15.46:0/2068463254 shutdown_connections 2024-01-21T08:18:51.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.220+0000 7fab361b4700 1 -- 172.21.15.46:0/2068463254 wait complete. 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab361b4700 1 Processor -- start 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab361b4700 1 -- start start 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab361b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30060870 0x7fab300f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab361b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab300f9930 0x7fab300f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab361b4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab300f5810 con 0x7fab300f9930 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab361b4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab300f5950 con 0x7fab30060870 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab300f9930 0x7fab300f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab300f9930 0x7fab300f52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41262/0 (socket says 172.21.15.46:41262) 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab2effd700 1 -- 172.21.15.46:0/328945149 learned_addr learned my addr 172.21.15.46:0/328945149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab2f7fe700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30060870 0x7fab300f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab2effd700 1 -- 172.21.15.46:0/328945149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30060870 msgr2=0x7fab300f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab2effd700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30060870 0x7fab300f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.221+0000 7fab2effd700 1 -- 172.21.15.46:0/328945149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab18005040 con 0x7fab300f9930 2024-01-21T08:18:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.222+0000 7fab2effd700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab300f9930 0x7fab300f52b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fab18000c00 tx=0x7fab1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.222+0000 7fab27fff700 1 -- 172.21.15.46:0/328945149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1800e070 con 0x7fab300f9930 2024-01-21T08:18:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.222+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab300f5bd0 con 0x7fab300f9930 2024-01-21T08:18:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.222+0000 7fab27fff700 1 -- 172.21.15.46:0/328945149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab18007ba0 con 0x7fab300f9930 2024-01-21T08:18:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.222+0000 7fab27fff700 1 -- 172.21.15.46:0/328945149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab18019070 con 0x7fab300f9930 2024-01-21T08:18:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.222+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab300f60b0 con 0x7fab300f9930 2024-01-21T08:18:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.223+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab30045ad0 con 0x7fab300f9930 2024-01-21T08:18:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.223+0000 7fab27fff700 1 -- 172.21.15.46:0/328945149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fab18014420 con 0x7fab300f9930 2024-01-21T08:18:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.223+0000 7fab27fff700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab1c05c2a0 0x7fab1c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.223+0000 7fab27fff700 1 -- 172.21.15.46:0/328945149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fab18076df0 con 0x7fab300f9930 2024-01-21T08:18:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.224+0000 7fab2f7fe700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab1c05c2a0 0x7fab1c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.224+0000 7fab2f7fe700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab1c05c2a0 0x7fab1c05e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fab300f6bc0 tx=0x7fab2000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:51.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.227+0000 7fab27fff700 1 -- 172.21.15.46:0/328945149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab18049df0 con 0x7fab300f9930 2024-01-21T08:18:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.387+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fab30027a00 con 0x7fab300f9930 2024-01-21T08:18:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.387+0000 7fab27fff700 1 -- 172.21.15.46:0/328945149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fab18007d10 con 0x7fab300f9930 2024-01-21T08:18:51.389 INFO:teuthology.orchestra.run.smithi046.stdout:150323855370 2024-01-21T08:18:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab1c05c2a0 msgr2=0x7fab1c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab1c05c2a0 0x7fab1c05e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fab300f6bc0 tx=0x7fab2000b040).stop 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab300f9930 msgr2=0x7fab300f52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab300f9930 0x7fab300f52b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fab18000c00 tx=0x7fab1800b040).stop 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 shutdown_connections 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab1c05c2a0 0x7fab1c05e760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30060870 0x7fab300f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 --2- 172.21.15.46:0/328945149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab300f9930 0x7fab300f52b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 >> 172.21.15.46:0/328945149 conn(0x7fab300f0e50 msgr2=0x7fab300f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 shutdown_connections 2024-01-21T08:18:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:51.389+0000 7fab361b4700 1 -- 172.21.15.46:0/328945149 wait complete. 2024-01-21T08:18:51.692 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:50.099948+0000 mgr.smithi046.lmxnwc (mgr.14164) 452 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:51.692 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:51.388040+0000 mon.smithi046 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.46:0/328945149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-21T08:18:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:50.099948+0000 mgr.smithi046.lmxnwc (mgr.14164) 452 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:51.388040+0000 mon.smithi046 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.46:0/328945149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-21T08:18:51.768 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855368 got 150323855370 for osd.5 2024-01-21T08:18:51.768 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-01-21T08:18:51.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph pg dump --format=json 2024-01-21T08:18:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:52.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 -- 172.21.15.46:0/20404839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 msgr2=0x7f22bc0f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:52.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 --2- 172.21.15.46:0/20404839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc0f5ae0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f22ac004660 tx=0x7f22ac009b30).stop 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 -- 172.21.15.46:0/20404839 shutdown_connections 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 --2- 172.21.15.46:0/20404839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22bc0f6960 0x7f22bc0f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 --2- 172.21.15.46:0/20404839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc0f5ae0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 -- 172.21.15.46:0/20404839 >> 172.21.15.46:0/20404839 conn(0x7f22bc0f0e20 msgr2=0x7f22bc0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 -- 172.21.15.46:0/20404839 shutdown_connections 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.964+0000 7f22c11c8700 1 -- 172.21.15.46:0/20404839 wait complete. 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22c11c8700 1 Processor -- start 2024-01-21T08:18:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22c11c8700 1 -- start start 2024-01-21T08:18:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22c11c8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc109d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22c11c8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22bc0f6960 0x7f22bc10c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22c11c8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc060920 con 0x7f22bc0f56c0 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22c11c8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc060a60 con 0x7f22bc0f6960 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22bad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22bad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc109d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41274/0 (socket says 172.21.15.46:41274) 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22bad9d700 1 -- 172.21.15.46:0/1255097463 learned_addr learned my addr 172.21.15.46:0/1255097463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.965+0000 7f22ba59c700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22bc0f6960 0x7f22bc10c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22bad9d700 1 -- 172.21.15.46:0/1255097463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22bc0f6960 msgr2=0x7f22bc10c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22bad9d700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22bc0f6960 0x7f22bc10c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22bad9d700 1 -- 172.21.15.46:0/1255097463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac005040 con 0x7f22bc0f56c0 2024-01-21T08:18:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22ba59c700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22bc0f6960 0x7f22bc10c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22bad9d700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc109d70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f22ac004e30 tx=0x7f22ac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22b37fe700 1 -- 172.21.15.46:0/1255097463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22ac018070 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22b37fe700 1 -- 172.21.15.46:0/1255097463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22ac013410 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc10c7c0 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22b37fe700 1 -- 172.21.15.46:0/1255097463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22ac01c3c0 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.966+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc10ccf0 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.967+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc059100 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.967+0000 7f22b37fe700 1 -- 172.21.15.46:0/1255097463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f22ac01d420 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.968+0000 7f22b37fe700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f22a805c2a0 0x7f22a805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.968+0000 7f22b37fe700 1 -- 172.21.15.46:0/1255097463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f22ac077700 con 0x7f22bc0f56c0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.968+0000 7f22ba59c700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f22a805c2a0 0x7f22a805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.968+0000 7f22ba59c700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f22a805c2a0 0x7f22a805e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f22a4009760 tx=0x7f22a4009c60).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:52.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:52.972+0000 7f22b37fe700 1 -- 172.21.15.46:0/1255097463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22ac04a780 con 0x7f22bc0f56c0 2024-01-21T08:18:53.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.126+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f22bc0fdfd0 con 0x7f22a805c2a0 2024-01-21T08:18:53.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.127+0000 7f22b37fe700 1 -- 172.21.15.46:0/1255097463 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24118 (secure 0 0 0) 0x7f22bc0fdfd0 con 0x7f22a805c2a0 2024-01-21T08:18:53.129 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f22a805c2a0 msgr2=0x7f22a805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f22a805c2a0 0x7f22a805e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f22a4009760 tx=0x7f22a4009c60).stop 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 msgr2=0x7f22bc109d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc109d70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f22ac004e30 tx=0x7f22ac009b30).stop 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 shutdown_connections 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f22a805c2a0 0x7f22a805e760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22bc0f56c0 0x7f22bc109d70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 --2- 172.21.15.46:0/1255097463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22bc0f6960 0x7f22bc10c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 >> 172.21.15.46:0/1255097463 conn(0x7f22bc0f0e20 msgr2=0x7f22bc0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 shutdown_connections 2024-01-21T08:18:53.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:53.129+0000 7f22c11c8700 1 -- 172.21.15.46:0/1255097463 wait complete. 2024-01-21T08:18:53.132 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-21T08:18:53.500 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-01-21T08:18:52.100093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31084,"kb_used_data":1916,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439828,"statfs":{"total":575970213888,"available":575938383872,"internally_reserved":0,"allocated":1961984,"data_stored":735915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2024-01-21T08:18:17.444079+0000","last_change":"2024-01-21T08:18:04.476404+0000","last_active":"2024-01-21T08:18:17.444079+0000","last_peered":"2024-01-21T08:18:17.444079+0000","last_clean":"2024-01-21T08:18:17.444079+0000","last_became_active":"2024-01-21T08:18:04.476037+0000","last_became_peered":"2024-01-21T08:18:04.476037+0000","last_unstale":"2024-01-21T08:18:17.444079+0000","last_undegraded":"2024-01-21T08:18:17.444079+0000","last_fullsized":"2024-01-21T08:18:17.444079+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T08:15:22.069921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T08:15:22.069921+0000","last_clean_scrub_stamp":"2024-01-21T08:15:22.069921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 08:18:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55200000000000005,"15min":0.56299999999999994},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.38200000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44500000000000001}]},{"osd":1,"last update":"Sun Jan 21 08:18:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70999999999999996,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46700000000000003}]},{"osd":2,"last update":"Sun Jan 21 08:18:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.50800000000000001,"15min":0.51300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]}]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5192,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739960,"statfs":{"total":95995035648,"available":95989719040,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 08:17:51 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36399999999999999},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.307},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.311}]},{"osd":1,"last update":"Sun Jan 21 08:17:51 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.253},{"interface":"front","average":{"1min":0.379,"5min":0.371,"15min":0.38500000000000001},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.49399999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.27700000000000002}]},{"osd":3,"last update":"Sun Jan 21 08:18:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.55600000000000005},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.66100000000000003,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5320,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739832,"statfs":{"total":95995035648,"available":95989587968,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 08:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.24099999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.58699999999999997},"last":0.372},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35999999999999999,"15min":0.36299999999999999},"min":{"1min":0.22800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.41499999999999998}]},{"osd":2,"last update":"Sun Jan 21 08:17:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.26200000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.38200000000000001}]},{"osd":3,"last update":"Sun Jan 21 08:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.52200000000000002,"15min":0.53400000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67200000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5256,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739896,"statfs":{"total":95995035648,"available":95989653504,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 08:18:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35699999999999998,"15min":0.375},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53000000000000003,"5min":0.53800000000000003,"15min":1.323},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.623},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":8.9160000000000004},"last":0.26100000000000001}]},{"osd":2,"last update":"Sun Jan 21 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.375,"15min":0.373},"min":{"1min":0.26800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.49399999999999999,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40100000000000002,"15min":0.38400000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.49099999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.29699999999999999}]},{"osd":3,"last update":"Sun Jan 21 08:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50600000000000001,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65100000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T08:18:53.501 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph pg dump --format=json 2024-01-21T08:18:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:53.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:52.100260+0000 mgr.smithi046.lmxnwc (mgr.14164) 453 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:53.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:52.100260+0000 mgr.smithi046.lmxnwc (mgr.14164) 453 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 -- 172.21.15.46:0/3522888499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 msgr2=0x7fad900f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 --2- 172.21.15.46:0/3522888499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad900f5ae0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fad80004660 tx=0x7fad80009b30).stop 2024-01-21T08:18:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 -- 172.21.15.46:0/3522888499 shutdown_connections 2024-01-21T08:18:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 --2- 172.21.15.46:0/3522888499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad900f6960 0x7fad900f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 --2- 172.21.15.46:0/3522888499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad900f5ae0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 -- 172.21.15.46:0/3522888499 >> 172.21.15.46:0/3522888499 conn(0x7fad900f0e20 msgr2=0x7fad900f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 -- 172.21.15.46:0/3522888499 shutdown_connections 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.672+0000 7fad96509700 1 -- 172.21.15.46:0/3522888499 wait complete. 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad96509700 1 Processor -- start 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad96509700 1 -- start start 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad96509700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad90105fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad96509700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad900f6960 0x7fad901064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad96509700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad90106a70 con 0x7fad900f56c0 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad96509700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad90106bb0 con 0x7fad900f6960 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad8ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad90105fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad8ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad90105fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41302/0 (socket says 172.21.15.46:41302) 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad8ffff700 1 -- 172.21.15.46:0/21457997 learned_addr learned my addr 172.21.15.46:0/21457997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad8f7fe700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad900f6960 0x7fad901064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad8ffff700 1 -- 172.21.15.46:0/21457997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad900f6960 msgr2=0x7fad901064c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad8ffff700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad900f6960 0x7fad901064c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.673+0000 7fad8ffff700 1 -- 172.21.15.46:0/21457997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad80005040 con 0x7fad900f56c0 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.674+0000 7fad8ffff700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad90105fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fad80004e30 tx=0x7fad80009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.674+0000 7fad8cff9700 1 -- 172.21.15.46:0/21457997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad80018070 con 0x7fad900f56c0 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.674+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad901074f0 con 0x7fad900f56c0 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.674+0000 7fad8cff9700 1 -- 172.21.15.46:0/21457997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad80013410 con 0x7fad900f56c0 2024-01-21T08:18:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.674+0000 7fad8cff9700 1 -- 172.21.15.46:0/21457997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad8001de20 con 0x7fad900f56c0 2024-01-21T08:18:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.674+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad90107a50 con 0x7fad900f56c0 2024-01-21T08:18:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.675+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad90045ad0 con 0x7fad900f56c0 2024-01-21T08:18:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.675+0000 7fad8cff9700 1 -- 172.21.15.46:0/21457997 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fad8001d420 con 0x7fad900f56c0 2024-01-21T08:18:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.675+0000 7fad8cff9700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad7805c2a0 0x7fad7805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.676+0000 7fad8cff9700 1 -- 172.21.15.46:0/21457997 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fad800777b0 con 0x7fad900f56c0 2024-01-21T08:18:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.676+0000 7fad8f7fe700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad7805c2a0 0x7fad7805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:54.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.676+0000 7fad8f7fe700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad7805c2a0 0x7fad7805e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fad84004c40 tx=0x7fad8400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:54.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.680+0000 7fad8cff9700 1 -- 172.21.15.46:0/21457997 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad8004a7b0 con 0x7fad900f56c0 2024-01-21T08:18:54.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:53.127211+0000 mgr.smithi046.lmxnwc (mgr.14164) 454 : audit [DBG] from='client.14572 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T08:18:54.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:53.127211+0000 mgr.smithi046.lmxnwc (mgr.14164) 454 : audit [DBG] from='client.14572 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T08:18:54.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.835+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 --> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad900fdfd0 con 0x7fad7805c2a0 2024-01-21T08:18:54.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.836+0000 7fad8cff9700 1 -- 172.21.15.46:0/21457997 <== mgr.14164 v2:172.21.15.46:6800/559524926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24146 (secure 0 0 0) 0x7fad900fdfd0 con 0x7fad7805c2a0 2024-01-21T08:18:54.838 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:54.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad7805c2a0 msgr2=0x7fad7805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:54.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad7805c2a0 0x7fad7805e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fad84004c40 tx=0x7fad8400b040).stop 2024-01-21T08:18:54.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 msgr2=0x7fad90105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:54.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad90105fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fad80004e30 tx=0x7fad80009b30).stop 2024-01-21T08:18:54.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 shutdown_connections 2024-01-21T08:18:54.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad7805c2a0 0x7fad7805e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:54.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad900f56c0 0x7fad90105fd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:54.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 --2- 172.21.15.46:0/21457997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad900f6960 0x7fad901064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:54.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 >> 172.21.15.46:0/21457997 conn(0x7fad900f0e20 msgr2=0x7fad900fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:54.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 shutdown_connections 2024-01-21T08:18:54.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:54.838+0000 7fad96509700 1 -- 172.21.15.46:0/21457997 wait complete. 2024-01-21T08:18:54.840 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-21T08:18:55.220 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-01-21T08:18:54.100444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31084,"kb_used_data":1916,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439828,"statfs":{"total":575970213888,"available":575938383872,"internally_reserved":0,"allocated":1961984,"data_stored":735915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2024-01-21T08:18:17.444079+0000","last_change":"2024-01-21T08:18:04.476404+0000","last_active":"2024-01-21T08:18:17.444079+0000","last_peered":"2024-01-21T08:18:17.444079+0000","last_clean":"2024-01-21T08:18:17.444079+0000","last_became_active":"2024-01-21T08:18:04.476037+0000","last_became_peered":"2024-01-21T08:18:04.476037+0000","last_unstale":"2024-01-21T08:18:17.444079+0000","last_undegraded":"2024-01-21T08:18:17.444079+0000","last_fullsized":"2024-01-21T08:18:17.444079+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T08:15:22.069921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T08:15:22.069921+0000","last_clean_scrub_stamp":"2024-01-21T08:15:22.069921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 08:18:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55200000000000005,"15min":0.56299999999999994},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.38200000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.69599999999999995}]},{"osd":1,"last update":"Sun Jan 21 08:18:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70999999999999996,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.60299999999999998}]},{"osd":2,"last update":"Sun Jan 21 08:18:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.50800000000000001,"15min":0.51300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]},{"osd":2,"up_from":17,"seq":73014444122,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5192,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739960,"statfs":{"total":95995035648,"available":95989719040,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 08:17:51 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36399999999999999},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.32200000000000001}]},{"osd":1,"last update":"Sun Jan 21 08:17:51 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.251},{"interface":"front","average":{"1min":0.379,"5min":0.371,"15min":0.38500000000000001},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.49399999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jan 21 08:18:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.55600000000000005},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.66100000000000003,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5320,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739832,"statfs":{"total":95995035648,"available":95989587968,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 08:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.24099999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.58699999999999997},"last":0.372},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35999999999999999,"15min":0.36299999999999999},"min":{"1min":0.22800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.41499999999999998}]},{"osd":2,"last update":"Sun Jan 21 08:17:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.26200000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.38200000000000001}]},{"osd":3,"last update":"Sun Jan 21 08:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.52200000000000002,"15min":0.53400000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67200000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5256,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739896,"statfs":{"total":95995035648,"available":95989653504,"internally_reserved":0,"allocated":327680,"data_stored":122705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 08:18:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35699999999999998,"15min":0.375},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53000000000000003,"5min":0.53800000000000003,"15min":1.323},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.623},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":8.9160000000000004},"last":0.27400000000000002}]},{"osd":2,"last update":"Sun Jan 21 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.375,"15min":0.373},"min":{"1min":0.26800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.49399999999999999,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40100000000000002,"15min":0.38400000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.49099999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Jan 21 08:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50600000000000001,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65100000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T08:18:55.221 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-01-21T08:18:55.221 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-21T08:18:55.221 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-01-21T08:18:55.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:18:55.452 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:18:55.491 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:54.100633+0000 mgr.smithi046.lmxnwc (mgr.14164) 455 : cluster [DBG] pgmap v433: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:54.100633+0000 mgr.smithi046.lmxnwc (mgr.14164) 455 : cluster [DBG] pgmap v433: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 -- 172.21.15.46:0/1515068224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 msgr2=0x7f3f300f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 --2- 172.21.15.46:0/1515068224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f300f6dc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f3f20004660 tx=0x7f3f20009b30).stop 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 -- 172.21.15.46:0/1515068224 shutdown_connections 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 --2- 172.21.15.46:0/1515068224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f300f6dc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 --2- 172.21.15.46:0/1515068224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f300f56c0 0x7f3f300f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 -- 172.21.15.46:0/1515068224 >> 172.21.15.46:0/1515068224 conn(0x7f3f300f0e20 msgr2=0x7f3f300f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 -- 172.21.15.46:0/1515068224 shutdown_connections 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.418+0000 7f3f369df700 1 -- 172.21.15.46:0/1515068224 wait complete. 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f369df700 1 Processor -- start 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f369df700 1 -- start start 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f369df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f300f56c0 0x7f3f301070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f369df700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f301075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f369df700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30107b70 con 0x7f3f300f6960 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f369df700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30107cb0 con 0x7f3f300f56c0 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f301075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f301075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41322/0 (socket says 172.21.15.46:41322) 2024-01-21T08:18:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2f7fe700 1 -- 172.21.15.46:0/593050829 learned_addr learned my addr 172.21.15.46:0/593050829 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:18:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2ffff700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f300f56c0 0x7f3f301070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2f7fe700 1 -- 172.21.15.46:0/593050829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f300f56c0 msgr2=0x7f3f301070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2f7fe700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f300f56c0 0x7f3f301070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2f7fe700 1 -- 172.21.15.46:0/593050829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20005040 con 0x7f3f300f6960 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.419+0000 7f3f2f7fe700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f301075c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f3f20009830 tx=0x7f3f20009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.420+0000 7f3f2cff9700 1 -- 172.21.15.46:0/593050829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f20018070 con 0x7f3f300f6960 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.420+0000 7f3f2cff9700 1 -- 172.21.15.46:0/593050829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f20013410 con 0x7f3f300f6960 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.420+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f301085f0 con 0x7f3f300f6960 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.420+0000 7f3f2cff9700 1 -- 172.21.15.46:0/593050829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f2001c400 con 0x7f3f300f6960 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.420+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f30108b50 con 0x7f3f300f6960 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.421+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f30045ad0 con 0x7f3f300f6960 2024-01-21T08:18:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.421+0000 7f3f2cff9700 1 -- 172.21.15.46:0/593050829 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3f2001c670 con 0x7f3f300f6960 2024-01-21T08:18:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.422+0000 7f3f2cff9700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f1c05c2a0 0x7f3f1c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:18:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.422+0000 7f3f2cff9700 1 -- 172.21.15.46:0/593050829 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3f20077950 con 0x7f3f300f6960 2024-01-21T08:18:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.422+0000 7f3f2ffff700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f1c05c2a0 0x7f3f1c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:18:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.422+0000 7f3f2ffff700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f1c05c2a0 0x7f3f1c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3f300f67c0 tx=0x7f3f18009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:18:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.428+0000 7f3f2cff9700 1 -- 172.21.15.46:0/593050829 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f2004a9d0 con 0x7f3f300f6960 2024-01-21T08:18:56.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:54.835936+0000 mgr.smithi046.lmxnwc (mgr.14164) 456 : audit [DBG] from='client.14576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T08:18:56.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.651+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3f300fdf00 con 0x7f3f300f6960 2024-01-21T08:18:56.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.651+0000 7f3f2cff9700 1 -- 172.21.15.46:0/593050829 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f3f2004dff0 con 0x7f3f300f6960 2024-01-21T08:18:56.653 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:18:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f1c05c2a0 msgr2=0x7f3f1c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f1c05c2a0 0x7f3f1c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3f300f67c0 tx=0x7f3f18009040).stop 2024-01-21T08:18:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 msgr2=0x7f3f301075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f301075c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f3f20009830 tx=0x7f3f20009b30).stop 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 shutdown_connections 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f1c05c2a0 0x7f3f1c05e760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f300f56c0 0x7f3f301070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 --2- 172.21.15.46:0/593050829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f300f6960 0x7f3f301075c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 >> 172.21.15.46:0/593050829 conn(0x7f3f300f0e20 msgr2=0x7f3f300603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 shutdown_connections 2024-01-21T08:18:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:18:56.653+0000 7f3f369df700 1 -- 172.21.15.46:0/593050829 wait complete. 2024-01-21T08:18:56.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:54.835936+0000 mgr.smithi046.lmxnwc (mgr.14164) 456 : audit [DBG] from='client.14576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T08:18:57.033 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:18:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:56.100936+0000 mgr.smithi046.lmxnwc (mgr.14164) 457 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:18:56.652060+0000 mon.smithi046 (mon.0) 500 : audit [DBG] from='client.? 172.21.15.46:0/593050829' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:18:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:56.100936+0000 mgr.smithi046.lmxnwc (mgr.14164) 457 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:18:56.652060+0000 mon.smithi046 (mon.0) 500 : audit [DBG] from='client.? 172.21.15.46:0/593050829' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:18:59.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:18:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:18:58.101464+0000 mgr.smithi046.lmxnwc (mgr.14164) 458 : cluster [DBG] pgmap v435: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:18:59.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:18:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:18:58.101464+0000 mgr.smithi046.lmxnwc (mgr.14164) 458 : cluster [DBG] pgmap v435: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:00.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:00.101826+0000 mgr.smithi046.lmxnwc (mgr.14164) 459 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:00.101826+0000 mgr.smithi046.lmxnwc (mgr.14164) 459 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 -- 172.21.15.46:0/2151544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58180f9920 msgr2=0x7f58180fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 --2- 172.21.15.46:0/2151544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58180f9920 0x7f58180fbd70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f580c004660 tx=0x7f580c009b30).stop 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 -- 172.21.15.46:0/2151544 shutdown_connections 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 --2- 172.21.15.46:0/2151544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58180f9920 0x7f58180fbd70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 --2- 172.21.15.46:0/2151544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5818060870 0x7f58180f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 -- 172.21.15.46:0/2151544 >> 172.21.15.46:0/2151544 conn(0x7f58180f0e20 msgr2=0x7f58180f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 -- 172.21.15.46:0/2151544 shutdown_connections 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.275+0000 7f581d082700 1 -- 172.21.15.46:0/2151544 wait complete. 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.276+0000 7f581d082700 1 Processor -- start 2024-01-21T08:19:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.276+0000 7f581d082700 1 -- start start 2024-01-21T08:19:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.276+0000 7f581d082700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818060870 0x7f58180f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.276+0000 7f581d082700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58180f9920 0x7f58180f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.276+0000 7f581d082700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58180f5800 con 0x7f5818060870 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.276+0000 7f581d082700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58180f5940 con 0x7f58180f9920 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.276+0000 7f5816d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818060870 0x7f58180f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f5816d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818060870 0x7f58180f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47988/0 (socket says 172.21.15.46:47988) 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f5816d9d700 1 -- 172.21.15.46:0/3261731914 learned_addr learned my addr 172.21.15.46:0/3261731914 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f5816d9d700 1 -- 172.21.15.46:0/3261731914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58180f9920 msgr2=0x7f58180f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f581659c700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58180f9920 0x7f58180f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f5816d9d700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58180f9920 0x7f58180f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f5816d9d700 1 -- 172.21.15.46:0/3261731914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c005040 con 0x7f5818060870 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f5816d9d700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818060870 0x7f58180f6e20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f580800c790 tx=0x7f5808007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3261731914 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f580800e680 con 0x7f5818060870 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3261731914 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f580800ec90 con 0x7f5818060870 2024-01-21T08:19:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58180f5c20 con 0x7f5818060870 2024-01-21T08:19:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3261731914 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5808013d60 con 0x7f5818060870 2024-01-21T08:19:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.277+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581810f130 con 0x7f5818060870 2024-01-21T08:19:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.278+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5818045ad0 con 0x7f5818060870 2024-01-21T08:19:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.278+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3261731914 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f580801a060 con 0x7f5818060870 2024-01-21T08:19:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.279+0000 7f57ff7fe700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f580005c2a0 0x7f580005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.279+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3261731914 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5808075da0 con 0x7f5818060870 2024-01-21T08:19:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.279+0000 7f581659c700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f580005c2a0 0x7f580005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.279+0000 7f581659c700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f580005c2a0 0x7f580005e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f580c000c00 tx=0x7f580c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:01.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.283+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3261731914 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5808048d40 con 0x7f5818060870 2024-01-21T08:19:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.501+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f58180feff0 con 0x7f5818060870 2024-01-21T08:19:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.501+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3261731914 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f580804c360 con 0x7f5818060870 2024-01-21T08:19:01.503 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f580005c2a0 msgr2=0x7f580005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f580005c2a0 0x7f580005e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f580c000c00 tx=0x7f580c009b30).stop 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818060870 msgr2=0x7f58180f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818060870 0x7f58180f6e20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f580800c790 tx=0x7f5808007d70).stop 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 shutdown_connections 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f580005c2a0 0x7f580005e760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818060870 0x7f58180f6e20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 --2- 172.21.15.46:0/3261731914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58180f9920 0x7f58180f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 >> 172.21.15.46:0/3261731914 conn(0x7f58180f0e20 msgr2=0x7f58180f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 shutdown_connections 2024-01-21T08:19:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:01.503+0000 7f581d082700 1 -- 172.21.15.46:0/3261731914 wait complete. 2024-01-21T08:19:01.799 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:01.502315+0000 mon.smithi046 (mon.0) 501 : audit [DBG] from='client.? 172.21.15.46:0/3261731914' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:01.892 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:01.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:01.502315+0000 mon.smithi046 (mon.0) 501 : audit [DBG] from='client.? 172.21.15.46:0/3261731914' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:02.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:02.102161+0000 mgr.smithi046.lmxnwc (mgr.14164) 460 : cluster [DBG] pgmap v437: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:02.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:02.102161+0000 mgr.smithi046.lmxnwc (mgr.14164) 460 : cluster [DBG] pgmap v437: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:04.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:05.137 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:05.645 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:04.102676+0000 mgr.smithi046.lmxnwc (mgr.14164) 461 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:05.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:04.102676+0000 mgr.smithi046.lmxnwc (mgr.14164) 461 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:06.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 -- 172.21.15.46:0/2783701066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 msgr2=0x7f07c40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:06.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 --2- 172.21.15.46:0/2783701066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f5ae0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f07c0004660 tx=0x7f07c0009b30).stop 2024-01-21T08:19:06.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 -- 172.21.15.46:0/2783701066 shutdown_connections 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 --2- 172.21.15.46:0/2783701066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07c40f6960 0x7f07c40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 --2- 172.21.15.46:0/2783701066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f5ae0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 -- 172.21.15.46:0/2783701066 >> 172.21.15.46:0/2783701066 conn(0x7f07c40f0e20 msgr2=0x7f07c40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 -- 172.21.15.46:0/2783701066 shutdown_connections 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.132+0000 7f07cc193700 1 -- 172.21.15.46:0/2783701066 wait complete. 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07cc193700 1 Processor -- start 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07cc193700 1 -- start start 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07cc193700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07cc193700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07c40f6960 0x7f07c40f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:06.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07cc193700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c40f8290 con 0x7f07c40f56c0 2024-01-21T08:19:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07cc193700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c40f83d0 con 0x7f07c40f6960 2024-01-21T08:19:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07c9f2f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07c9f2f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48000/0 (socket says 172.21.15.46:48000) 2024-01-21T08:19:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07c9f2f700 1 -- 172.21.15.46:0/3393776212 learned_addr learned my addr 172.21.15.46:0/3393776212 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07c972e700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07c40f6960 0x7f07c40f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07c9f2f700 1 -- 172.21.15.46:0/3393776212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07c40f6960 msgr2=0x7f07c40f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07c9f2f700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07c40f6960 0x7f07c40f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.133+0000 7f07c9f2f700 1 -- 172.21.15.46:0/3393776212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0005040 con 0x7f07c40f56c0 2024-01-21T08:19:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.134+0000 7f07c9f2f700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f9820 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f07c0009f50 tx=0x7f07c0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.134+0000 7f07b67fc700 1 -- 172.21.15.46:0/3393776212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07c0018070 con 0x7f07c40f56c0 2024-01-21T08:19:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.134+0000 7f07b67fc700 1 -- 172.21.15.46:0/3393776212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07c0013410 con 0x7f07c40f56c0 2024-01-21T08:19:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.134+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c40f8650 con 0x7f07c40f56c0 2024-01-21T08:19:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.134+0000 7f07b67fc700 1 -- 172.21.15.46:0/3393776212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07c001c620 con 0x7f07c40f56c0 2024-01-21T08:19:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.134+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c410f110 con 0x7f07c40f56c0 2024-01-21T08:19:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.135+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c4045ad0 con 0x7f07c40f56c0 2024-01-21T08:19:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.135+0000 7f07b67fc700 1 -- 172.21.15.46:0/3393776212 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f07c001d420 con 0x7f07c40f56c0 2024-01-21T08:19:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.135+0000 7f07b67fc700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07b005c2a0 0x7f07b005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.135+0000 7f07b67fc700 1 -- 172.21.15.46:0/3393776212 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f07c00779f0 con 0x7f07c40f56c0 2024-01-21T08:19:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.136+0000 7f07c972e700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07b005c2a0 0x7f07b005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.136+0000 7f07c972e700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07b005c2a0 0x7f07b005e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f07b8006fd0 tx=0x7f07b8008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.140+0000 7f07b67fc700 1 -- 172.21.15.46:0/3393776212 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07c0079e60 con 0x7f07c40f56c0 2024-01-21T08:19:06.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.361+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f07c40fded0 con 0x7f07c40f56c0 2024-01-21T08:19:06.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.361+0000 7f07b67fc700 1 -- 172.21.15.46:0/3393776212 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f07c004a9f0 con 0x7f07c40f56c0 2024-01-21T08:19:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:06.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07b005c2a0 msgr2=0x7f07b005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:06.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07b005c2a0 0x7f07b005e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f07b8006fd0 tx=0x7f07b8008040).stop 2024-01-21T08:19:06.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 msgr2=0x7f07c40f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:06.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f9820 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f07c0009f50 tx=0x7f07c0009b30).stop 2024-01-21T08:19:06.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 shutdown_connections 2024-01-21T08:19:06.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f07b005c2a0 0x7f07b005e760 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:06.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07c40f56c0 0x7f07c40f9820 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:06.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 --2- 172.21.15.46:0/3393776212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07c40f6960 0x7f07c40f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:06.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 >> 172.21.15.46:0/3393776212 conn(0x7f07c40f0e20 msgr2=0x7f07c40603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:06.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 shutdown_connections 2024-01-21T08:19:06.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:06.363+0000 7f07cc193700 1 -- 172.21.15.46:0/3393776212 wait complete. 2024-01-21T08:19:06.673 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:06.361817+0000 mon.smithi046 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.46:0/3393776212' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:06.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:06.361817+0000 mon.smithi046 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.46:0/3393776212' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:06.749 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:06.102979+0000 mgr.smithi046.lmxnwc (mgr.14164) 462 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:07.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:06.102979+0000 mgr.smithi046.lmxnwc (mgr.14164) 462 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:09.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:08.103330+0000 mgr.smithi046.lmxnwc (mgr.14164) 463 : cluster [DBG] pgmap v440: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:09.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:08.103330+0000 mgr.smithi046.lmxnwc (mgr.14164) 463 : cluster [DBG] pgmap v440: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:09.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:09.996 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 -- 172.21.15.46:0/953732170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f67b0 msgr2=0x7f75800f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 --2- 172.21.15.46:0/953732170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f67b0 0x7f75800f6c10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f7570004660 tx=0x7f7570009b30).stop 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 -- 172.21.15.46:0/953732170 shutdown_connections 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 --2- 172.21.15.46:0/953732170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f67b0 0x7f75800f6c10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 --2- 172.21.15.46:0/953732170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75800f5510 0x7f75800f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 -- 172.21.15.46:0/953732170 >> 172.21.15.46:0/953732170 conn(0x7f75800f0c70 msgr2=0x7f75800f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 -- 172.21.15.46:0/953732170 shutdown_connections 2024-01-21T08:19:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.953+0000 7f7584846700 1 -- 172.21.15.46:0/953732170 wait complete. 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f7584846700 1 Processor -- start 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f7584846700 1 -- start start 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f7584846700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f5510 0x7f7580109b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f7584846700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75800f67b0 0x7f758010c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f7584846700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7580060940 con 0x7f75800f5510 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f7584846700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7580060a80 con 0x7f75800f67b0 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f757f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f5510 0x7f7580109b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f757f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f5510 0x7f7580109b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50352/0 (socket says 172.21.15.46:50352) 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.954+0000 7f757f7fe700 1 -- 172.21.15.46:0/2339484919 learned_addr learned my addr 172.21.15.46:0/2339484919 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f757f7fe700 1 -- 172.21.15.46:0/2339484919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75800f67b0 msgr2=0x7f758010c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f757f7fe700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75800f67b0 0x7f758010c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:19:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f757f7fe700 1 -- 172.21.15.46:0/2339484919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7570005040 con 0x7f75800f5510 2024-01-21T08:19:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f757f7fe700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f5510 0x7f7580109b10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f757400e7c0 tx=0x7f757400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f7567fff700 1 -- 172.21.15.46:0/2339484919 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75740092e0 con 0x7f75800f5510 2024-01-21T08:19:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758010c5c0 con 0x7f75800f5510 2024-01-21T08:19:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f7567fff700 1 -- 172.21.15.46:0/2339484919 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f757400f040 con 0x7f75800f5510 2024-01-21T08:19:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f7567fff700 1 -- 172.21.15.46:0/2339484919 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7574014750 con 0x7f75800f5510 2024-01-21T08:19:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.955+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758010cb80 con 0x7f75800f5510 2024-01-21T08:19:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.956+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7580045ad0 con 0x7f75800f5510 2024-01-21T08:19:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.956+0000 7f7567fff700 1 -- 172.21.15.46:0/2339484919 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f757401b030 con 0x7f75800f5510 2024-01-21T08:19:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.956+0000 7f7567fff700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f756805c2b0 0x7f756805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.957+0000 7f7567fff700 1 -- 172.21.15.46:0/2339484919 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7574075c60 con 0x7f75800f5510 2024-01-21T08:19:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.957+0000 7f757effd700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f756805c2b0 0x7f756805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.957+0000 7f757effd700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f756805c2b0 0x7f756805e770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7570013530 tx=0x7f757000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:10.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:10.961+0000 7f7567fff700 1 -- 172.21.15.46:0/2339484919 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7574048c60 con 0x7f75800f5510 2024-01-21T08:19:11.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.179+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f75800fde20 con 0x7f75800f5510 2024-01-21T08:19:11.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.180+0000 7f7567fff700 1 -- 172.21.15.46:0/2339484919 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f7574019080 con 0x7f75800f5510 2024-01-21T08:19:11.181 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f756805c2b0 msgr2=0x7f756805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f756805c2b0 0x7f756805e770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7570013530 tx=0x7f757000b040).stop 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f5510 msgr2=0x7f7580109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f5510 0x7f7580109b10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f757400e7c0 tx=0x7f757400b040).stop 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 shutdown_connections 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f756805c2b0 0x7f756805e770 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75800f5510 0x7f7580109b10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 --2- 172.21.15.46:0/2339484919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75800f67b0 0x7f758010c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 >> 172.21.15.46:0/2339484919 conn(0x7f75800f0c70 msgr2=0x7f75800fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 shutdown_connections 2024-01-21T08:19:11.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:11.181+0000 7f7584846700 1 -- 172.21.15.46:0/2339484919 wait complete. 2024-01-21T08:19:11.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:10.103638+0000 mgr.smithi046.lmxnwc (mgr.14164) 464 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:11.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:11.180497+0000 mon.smithi046 (mon.0) 503 : audit [DBG] from='client.? 172.21.15.46:0/2339484919' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:11.557 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:10.103638+0000 mgr.smithi046.lmxnwc (mgr.14164) 464 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:11.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:11.180497+0000 mon.smithi046 (mon.0) 503 : audit [DBG] from='client.? 172.21.15.46:0/2339484919' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:12.103967+0000 mgr.smithi046.lmxnwc (mgr.14164) 465 : cluster [DBG] pgmap v442: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:12.103967+0000 mgr.smithi046.lmxnwc (mgr.14164) 465 : cluster [DBG] pgmap v442: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:14.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:15.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:14.104408+0000 mgr.smithi046.lmxnwc (mgr.14164) 466 : cluster [DBG] pgmap v443: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:15.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:14.104408+0000 mgr.smithi046.lmxnwc (mgr.14164) 466 : cluster [DBG] pgmap v443: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3848456296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f67e0 msgr2=0x7f74c80f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3848456296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f67e0 0x7f74c80f6c00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f74b8004660 tx=0x7f74b8009b30).stop 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3848456296 shutdown_connections 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3848456296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c80f7a80 0x7f74c80f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3848456296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f67e0 0x7f74c80f6c00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3848456296 >> 172.21.15.46:0/3848456296 conn(0x7f74c80f1f80 msgr2=0x7f74c80f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3848456296 shutdown_connections 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.793+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3848456296 wait complete. 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74ce2d3700 1 Processor -- start 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74ce2d3700 1 -- start start 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74ce2d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c80f67e0 0x7f74c8107020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74ce2d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f7a80 0x7f74c8107510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74ce2d3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8107ac0 con 0x7f74c80f7a80 2024-01-21T08:19:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74ce2d3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8107c00 con 0x7f74c80f67e0 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f7a80 0x7f74c8107510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f7a80 0x7f74c8107510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50356/0 (socket says 172.21.15.46:50356) 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.794+0000 7f74c77fe700 1 -- 172.21.15.46:0/3157150739 learned_addr learned my addr 172.21.15.46:0/3157150739 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c7fff700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c80f67e0 0x7f74c8107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c77fe700 1 -- 172.21.15.46:0/3157150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c80f67e0 msgr2=0x7f74c8107020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c77fe700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c80f67e0 0x7f74c8107020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c77fe700 1 -- 172.21.15.46:0/3157150739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b8005040 con 0x7f74c80f7a80 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c77fe700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f7a80 0x7f74c8107510 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f74bc00e7f0 tx=0x7f74bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c4ff9700 1 -- 172.21.15.46:0/3157150739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74bc0092e0 con 0x7f74c80f7a80 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c81085a0 con 0x7f74c80f7a80 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c4ff9700 1 -- 172.21.15.46:0/3157150739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74bc00f040 con 0x7f74c80f7a80 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74c4ff9700 1 -- 172.21.15.46:0/3157150739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74bc0146c0 con 0x7f74c80f7a80 2024-01-21T08:19:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.795+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c8108b60 con 0x7f74c80f7a80 2024-01-21T08:19:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.796+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c8045ad0 con 0x7f74c80f7a80 2024-01-21T08:19:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.796+0000 7f74c4ff9700 1 -- 172.21.15.46:0/3157150739 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f74bc01b030 con 0x7f74c80f7a80 2024-01-21T08:19:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.797+0000 7f74c4ff9700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f74b005c2a0 0x7f74b005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.797+0000 7f74c4ff9700 1 -- 172.21.15.46:0/3157150739 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f74bc075930 con 0x7f74c80f7a80 2024-01-21T08:19:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.797+0000 7f74c7fff700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f74b005c2a0 0x7f74b005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.797+0000 7f74c7fff700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f74b005c2a0 0x7f74b005e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f74b8005010 tx=0x7f74b800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:15.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:15.801+0000 7f74c4ff9700 1 -- 172.21.15.46:0/3157150739 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74bc048980 con 0x7f74c80f7a80 2024-01-21T08:19:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.019+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f74c8027a00 con 0x7f74c80f7a80 2024-01-21T08:19:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.020+0000 7f74c4ff9700 1 -- 172.21.15.46:0/3157150739 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f74bc019080 con 0x7f74c80f7a80 2024-01-21T08:19:16.021 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f74b005c2a0 msgr2=0x7f74b005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f74b005c2a0 0x7f74b005e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f74b8005010 tx=0x7f74b800b040).stop 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f7a80 msgr2=0x7f74c8107510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f7a80 0x7f74c8107510 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f74bc00e7f0 tx=0x7f74bc00b040).stop 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 shutdown_connections 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f74b005c2a0 0x7f74b005e760 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c80f67e0 0x7f74c8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 --2- 172.21.15.46:0/3157150739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c80f7a80 0x7f74c8107510 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 >> 172.21.15.46:0/3157150739 conn(0x7f74c80f1f80 msgr2=0x7f74c80fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 shutdown_connections 2024-01-21T08:19:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:16.021+0000 7f74ce2d3700 1 -- 172.21.15.46:0/3157150739 wait complete. 2024-01-21T08:19:16.398 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:16.020549+0000 mon.smithi046 (mon.0) 504 : audit [DBG] from='client.? 172.21.15.46:0/3157150739' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:16.104891+0000 mgr.smithi046.lmxnwc (mgr.14164) 467 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:16.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:16.020549+0000 mon.smithi046 (mon.0) 504 : audit [DBG] from='client.? 172.21.15.46:0/3157150739' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:16.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:16.104891+0000 mgr.smithi046.lmxnwc (mgr.14164) 467 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:19.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:19.433 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:18.105268+0000 mgr.smithi046.lmxnwc (mgr.14164) 468 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:19.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:18.105268+0000 mgr.smithi046.lmxnwc (mgr.14164) 468 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:20.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 -- 172.21.15.46:0/638689155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 msgr2=0x7f5f040f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:20.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 --2- 172.21.15.46:0/638689155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f040f6de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f5ef8004660 tx=0x7f5ef8009b30).stop 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 -- 172.21.15.46:0/638689155 shutdown_connections 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 --2- 172.21.15.46:0/638689155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f040f6de0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 --2- 172.21.15.46:0/638689155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f040f56e0 0x7f5f040f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 -- 172.21.15.46:0/638689155 >> 172.21.15.46:0/638689155 conn(0x7f5f040f0e20 msgr2=0x7f5f040f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 -- 172.21.15.46:0/638689155 shutdown_connections 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.543+0000 7f5f0942b700 1 -- 172.21.15.46:0/638689155 wait complete. 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f0942b700 1 Processor -- start 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f0942b700 1 -- start start 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f0942b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f040f56e0 0x7f5f041070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f0942b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f04107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f0942b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04107b40 con 0x7f5f040f6980 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f0942b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04107c80 con 0x7f5f040f56e0 2024-01-21T08:19:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f04107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f04107590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55510/0 (socket says 172.21.15.46:55510) 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f027fc700 1 -- 172.21.15.46:0/4178575018 learned_addr learned my addr 172.21.15.46:0/4178575018 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.544+0000 7f5f02ffd700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f040f56e0 0x7f5f041070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5f027fc700 1 -- 172.21.15.46:0/4178575018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f040f56e0 msgr2=0x7f5f041070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5f027fc700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f040f56e0 0x7f5f041070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5f027fc700 1 -- 172.21.15.46:0/4178575018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef8005040 con 0x7f5f040f6980 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5f027fc700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f04107590 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5ef8009830 tx=0x7f5ef8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5eeb7fe700 1 -- 172.21.15.46:0/4178575018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef8018070 con 0x7f5f040f6980 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5eeb7fe700 1 -- 172.21.15.46:0/4178575018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ef8013410 con 0x7f5f040f6980 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f041085c0 con 0x7f5f040f6980 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5eeb7fe700 1 -- 172.21.15.46:0/4178575018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef801c620 con 0x7f5f040f6980 2024-01-21T08:19:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.545+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f04108b20 con 0x7f5f040f6980 2024-01-21T08:19:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.546+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f04045ad0 con 0x7f5f040f6980 2024-01-21T08:19:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.546+0000 7f5eeb7fe700 1 -- 172.21.15.46:0/4178575018 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5ef801c890 con 0x7f5f040f6980 2024-01-21T08:19:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.547+0000 7f5eeb7fe700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5eec05c2a0 0x7f5eec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.547+0000 7f5eeb7fe700 1 -- 172.21.15.46:0/4178575018 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5ef8077970 con 0x7f5f040f6980 2024-01-21T08:19:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.547+0000 7f5f02ffd700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5eec05c2a0 0x7f5eec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.547+0000 7f5f02ffd700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5eec05c2a0 0x7f5eec05e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5f040f67e0 tx=0x7f5ef4009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:20.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.551+0000 7f5eeb7fe700 1 -- 172.21.15.46:0/4178575018 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ef804a990 con 0x7f5f040f6980 2024-01-21T08:19:20.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.770+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5f040fdf20 con 0x7f5f040f6980 2024-01-21T08:19:20.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.770+0000 7f5eeb7fe700 1 -- 172.21.15.46:0/4178575018 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f5ef804dfb0 con 0x7f5f040f6980 2024-01-21T08:19:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5eec05c2a0 msgr2=0x7f5eec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5eec05c2a0 0x7f5eec05e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5f040f67e0 tx=0x7f5ef4009040).stop 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 msgr2=0x7f5f04107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f04107590 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5ef8009830 tx=0x7f5ef8009b30).stop 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 shutdown_connections 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5eec05c2a0 0x7f5eec05e760 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f040f56e0 0x7f5f041070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 --2- 172.21.15.46:0/4178575018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040f6980 0x7f5f04107590 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 >> 172.21.15.46:0/4178575018 conn(0x7f5f040f0e20 msgr2=0x7f5f040603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 shutdown_connections 2024-01-21T08:19:20.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:20.772+0000 7f5f0942b700 1 -- 172.21.15.46:0/4178575018 wait complete. 2024-01-21T08:19:21.147 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:20.105578+0000 mgr.smithi046.lmxnwc (mgr.14164) 469 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:20.760149+0000 mon.smithi046 (mon.0) 505 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:19:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:20.760346+0000 mon.smithi046 (mon.0) 506 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:19:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:20.771276+0000 mon.smithi046 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.46:0/4178575018' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:20.105578+0000 mgr.smithi046.lmxnwc (mgr.14164) 469 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:20.760149+0000 mon.smithi046 (mon.0) 505 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:19:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:20.760346+0000 mon.smithi046 (mon.0) 506 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:19:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:20.771276+0000 mon.smithi046 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.46:0/4178575018' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:23.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:22.106006+0000 mgr.smithi046.lmxnwc (mgr.14164) 470 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:22.106006+0000 mgr.smithi046.lmxnwc (mgr.14164) 470 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:24.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:25.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 -- 172.21.15.46:0/1248801366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f6620 msgr2=0x7f09c40f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:25.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 --2- 172.21.15.46:0/1248801366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f6620 0x7f09c40f6a40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f09c0004660 tx=0x7f09c0009b30).stop 2024-01-21T08:19:25.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 -- 172.21.15.46:0/1248801366 shutdown_connections 2024-01-21T08:19:25.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 --2- 172.21.15.46:0/1248801366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c40f78c0 0x7f09c40f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:25.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 --2- 172.21.15.46:0/1248801366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f6620 0x7f09c40f6a40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:25.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 -- 172.21.15.46:0/1248801366 >> 172.21.15.46:0/1248801366 conn(0x7f09c40f1da0 msgr2=0x7f09c40f41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 -- 172.21.15.46:0/1248801366 shutdown_connections 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 -- 172.21.15.46:0/1248801366 wait complete. 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 Processor -- start 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.393+0000 7f09cab3a700 1 -- start start 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09cab3a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c40f6620 0x7f09c4106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09cab3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f78c0 0x7f09c41073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09cab3a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c4107960 con 0x7f09c40f78c0 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09cab3a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c4107aa0 con 0x7f09c40f6620 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9337700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f78c0 0x7f09c41073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9337700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f78c0 0x7f09c41073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55530/0 (socket says 172.21.15.46:55530) 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9337700 1 -- 172.21.15.46:0/4204717993 learned_addr learned my addr 172.21.15.46:0/4204717993 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9b38700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c40f6620 0x7f09c4106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9337700 1 -- 172.21.15.46:0/4204717993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c40f6620 msgr2=0x7f09c4106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9337700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c40f6620 0x7f09c4106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9337700 1 -- 172.21.15.46:0/4204717993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0005040 con 0x7f09c40f78c0 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09c9337700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f78c0 0x7f09c41073b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f09b4002700 tx=0x7f09b4009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09ba7fc700 1 -- 172.21.15.46:0/4204717993 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b4009320 con 0x7f09c40f78c0 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09ba7fc700 1 -- 172.21.15.46:0/4204717993 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09b400f040 con 0x7f09c40f78c0 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c4108440 con 0x7f09c40f78c0 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09ba7fc700 1 -- 172.21.15.46:0/4204717993 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b4014820 con 0x7f09c40f78c0 2024-01-21T08:19:25.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.394+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c4108a00 con 0x7f09c40f78c0 2024-01-21T08:19:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.395+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c4045ad0 con 0x7f09c40f78c0 2024-01-21T08:19:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.396+0000 7f09ba7fc700 1 -- 172.21.15.46:0/4204717993 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f09b40077b0 con 0x7f09c40f78c0 2024-01-21T08:19:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.396+0000 7f09ba7fc700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f09b005c2b0 0x7f09b005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.396+0000 7f09ba7fc700 1 -- 172.21.15.46:0/4204717993 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f09b4075e40 con 0x7f09c40f78c0 2024-01-21T08:19:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.396+0000 7f09c9b38700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f09b005c2b0 0x7f09b005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.396+0000 7f09c9b38700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f09b005c2b0 0x7f09b005e770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f09c0004e30 tx=0x7f09c0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:25.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.400+0000 7f09ba7fc700 1 -- 172.21.15.46:0/4204717993 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09b4048e40 con 0x7f09c40f78c0 2024-01-21T08:19:25.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:24.106416+0000 mgr.smithi046.lmxnwc (mgr.14164) 471 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:25.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.621+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f09c4027a00 con 0x7f09c40f78c0 2024-01-21T08:19:25.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.621+0000 7f09ba7fc700 1 -- 172.21.15.46:0/4204717993 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f09b404c460 con 0x7f09c40f78c0 2024-01-21T08:19:25.623 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:25.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f09b005c2b0 msgr2=0x7f09b005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:25.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f09b005c2b0 0x7f09b005e770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f09c0004e30 tx=0x7f09c0009b30).stop 2024-01-21T08:19:25.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f78c0 msgr2=0x7f09c41073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:25.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f78c0 0x7f09c41073b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f09b4002700 tx=0x7f09b4009e00).stop 2024-01-21T08:19:25.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 shutdown_connections 2024-01-21T08:19:25.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f09b005c2b0 0x7f09b005e770 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:25.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c40f6620 0x7f09c4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:25.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 --2- 172.21.15.46:0/4204717993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c40f78c0 0x7f09c41073b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:25.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 >> 172.21.15.46:0/4204717993 conn(0x7f09c40f1da0 msgr2=0x7f09c40fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:25.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 shutdown_connections 2024-01-21T08:19:25.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:25.623+0000 7f09cab3a700 1 -- 172.21.15.46:0/4204717993 wait complete. 2024-01-21T08:19:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:24.106416+0000 mgr.smithi046.lmxnwc (mgr.14164) 471 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:26.006 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:25.622091+0000 mon.smithi046 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.46:0/4204717993' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:26.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:25.622091+0000 mon.smithi046 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.46:0/4204717993' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:27.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:26.106743+0000 mgr.smithi046.lmxnwc (mgr.14164) 472 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:27.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:26.106743+0000 mgr.smithi046.lmxnwc (mgr.14164) 472 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:29.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:29.250 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:29.717 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:28.107222+0000 mgr.smithi046.lmxnwc (mgr.14164) 473 : cluster [DBG] pgmap v450: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:29.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:28.107222+0000 mgr.smithi046.lmxnwc (mgr.14164) 473 : cluster [DBG] pgmap v450: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:30.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 -- 172.21.15.46:0/210268257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f7a70 msgr2=0x7fbe200f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:30.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 --2- 172.21.15.46:0/210268257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f7a70 0x7fbe200f7ed0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fbe14004660 tx=0x7fbe14009b30).stop 2024-01-21T08:19:30.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 -- 172.21.15.46:0/210268257 shutdown_connections 2024-01-21T08:19:30.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 --2- 172.21.15.46:0/210268257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f7a70 0x7fbe200f7ed0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:30.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 --2- 172.21.15.46:0/210268257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe200f67d0 0x7fbe200f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:19:30.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 -- 172.21.15.46:0/210268257 >> 172.21.15.46:0/210268257 conn(0x7fbe200f1f50 msgr2=0x7fbe200f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:30.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 -- 172.21.15.46:0/210268257 shutdown_connections 2024-01-21T08:19:30.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.183+0000 7fbe2688e700 1 -- 172.21.15.46:0/210268257 wait complete. 2024-01-21T08:19:30.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe2688e700 1 Processor -- start 2024-01-21T08:19:30.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe2688e700 1 -- start start 2024-01-21T08:19:30.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe2688e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f67d0 0x7fbe201070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:30.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe2688e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe200f7a70 0x7fbe201075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe2688e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20107bf0 con 0x7fbe200f67d0 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe2688e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe201084a0 con 0x7fbe200f7a70 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe1ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f67d0 0x7fbe201070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe1ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f67d0 0x7fbe201070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55554/0 (socket says 172.21.15.46:55554) 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.184+0000 7fbe1ffff700 1 -- 172.21.15.46:0/3268083206 learned_addr learned my addr 172.21.15.46:0/3268083206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1ffff700 1 -- 172.21.15.46:0/3268083206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe200f7a70 msgr2=0x7fbe201075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1f7fe700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe200f7a70 0x7fbe201075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1ffff700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe200f7a70 0x7fbe201075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1ffff700 1 -- 172.21.15.46:0/3268083206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14005040 con 0x7fbe200f67d0 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1ffff700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f67d0 0x7fbe201070c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbe100093f0 tx=0x7fbe10006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1cff9700 1 -- 172.21.15.46:0/3268083206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe10009ea0 con 0x7fbe200f67d0 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1cff9700 1 -- 172.21.15.46:0/3268083206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe10013af0 con 0x7fbe200f67d0 2024-01-21T08:19:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe20108780 con 0x7fbe200f67d0 2024-01-21T08:19:30.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe1cff9700 1 -- 172.21.15.46:0/3268083206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1000dad0 con 0x7fbe200f67d0 2024-01-21T08:19:30.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.185+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe20108d40 con 0x7fbe200f67d0 2024-01-21T08:19:30.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.186+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe20059100 con 0x7fbe200f67d0 2024-01-21T08:19:30.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.186+0000 7fbe1cff9700 1 -- 172.21.15.46:0/3268083206 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbe1001a060 con 0x7fbe200f67d0 2024-01-21T08:19:30.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.187+0000 7fbe1cff9700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe0805c2a0 0x7fbe0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:30.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.187+0000 7fbe1cff9700 1 -- 172.21.15.46:0/3268083206 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbe10075b40 con 0x7fbe200f67d0 2024-01-21T08:19:30.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.187+0000 7fbe1f7fe700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe0805c2a0 0x7fbe0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:30.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.187+0000 7fbe1f7fe700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe0805c2a0 0x7fbe0805e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbe14009d60 tx=0x7fbe14009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:30.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.191+0000 7fbe1cff9700 1 -- 172.21.15.46:0/3268083206 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe10048b60 con 0x7fbe200f67d0 2024-01-21T08:19:30.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.410+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe200ff0a0 con 0x7fbe200f67d0 2024-01-21T08:19:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.410+0000 7fbe1cff9700 1 -- 172.21.15.46:0/3268083206 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fbe1004c180 con 0x7fbe200f67d0 2024-01-21T08:19:30.412 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe0805c2a0 msgr2=0x7fbe0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe0805c2a0 0x7fbe0805e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbe14009d60 tx=0x7fbe14009b30).stop 2024-01-21T08:19:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f67d0 msgr2=0x7fbe201070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f67d0 0x7fbe201070c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbe100093f0 tx=0x7fbe10006ec0).stop 2024-01-21T08:19:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 shutdown_connections 2024-01-21T08:19:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbe0805c2a0 0x7fbe0805e760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe200f67d0 0x7fbe201070c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 --2- 172.21.15.46:0/3268083206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe200f7a70 0x7fbe201075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 >> 172.21.15.46:0/3268083206 conn(0x7fbe200f1f50 msgr2=0x7fbe200fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 shutdown_connections 2024-01-21T08:19:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:30.412+0000 7fbe2688e700 1 -- 172.21.15.46:0/3268083206 wait complete. 2024-01-21T08:19:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:30.411081+0000 mon.smithi046 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.46:0/3268083206' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:30.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:30.411081+0000 mon.smithi046 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.46:0/3268083206' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:30.787 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:30.107557+0000 mgr.smithi046.lmxnwc (mgr.14164) 474 : cluster [DBG] pgmap v451: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:30.107557+0000 mgr.smithi046.lmxnwc (mgr.14164) 474 : cluster [DBG] pgmap v451: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:32.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:32.107898+0000 mgr.smithi046.lmxnwc (mgr.14164) 475 : cluster [DBG] pgmap v452: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:32.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:32 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:32.107898+0000 mgr.smithi046.lmxnwc (mgr.14164) 475 : cluster [DBG] pgmap v452: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:33.788 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:34.033 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:35.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 -- 172.21.15.46:0/676415843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f6960 msgr2=0x7f21d00f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 --2- 172.21.15.46:0/676415843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f6960 0x7f21d00f6dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f21c4004660 tx=0x7f21c4009b30).stop 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 -- 172.21.15.46:0/676415843 shutdown_connections 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 --2- 172.21.15.46:0/676415843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f6960 0x7f21d00f6dc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 --2- 172.21.15.46:0/676415843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d00f56c0 0x7f21d00f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 -- 172.21.15.46:0/676415843 >> 172.21.15.46:0/676415843 conn(0x7f21d00f0e20 msgr2=0x7f21d00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 -- 172.21.15.46:0/676415843 shutdown_connections 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:34.999+0000 7f21d674a700 1 -- 172.21.15.46:0/676415843 wait complete. 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21d674a700 1 Processor -- start 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21d674a700 1 -- start start 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21d674a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f56c0 0x7f21d00f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21d674a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d00f6960 0x7f21d00f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21d674a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00f8200 con 0x7f21d00f56c0 2024-01-21T08:19:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21d674a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00f8340 con 0x7f21d00f6960 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f56c0 0x7f21d00f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f56c0 0x7f21d00f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47468/0 (socket says 172.21.15.46:47468) 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21cffff700 1 -- 172.21.15.46:0/3074738731 learned_addr learned my addr 172.21.15.46:0/3074738731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.000+0000 7f21cf7fe700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d00f6960 0x7f21d00f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21cffff700 1 -- 172.21.15.46:0/3074738731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d00f6960 msgr2=0x7f21d00f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21cffff700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d00f6960 0x7f21d00f7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21cffff700 1 -- 172.21.15.46:0/3074738731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4005040 con 0x7f21d00f56c0 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21cf7fe700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d00f6960 0x7f21d00f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21cffff700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f56c0 0x7f21d00f9820 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f21c000c530 tx=0x7f21c0007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21ccff9700 1 -- 172.21.15.46:0/3074738731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21c000f650 con 0x7f21d00f56c0 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21ccff9700 1 -- 172.21.15.46:0/3074738731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21c000fc60 con 0x7f21d00f56c0 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d00f8620 con 0x7f21d00f56c0 2024-01-21T08:19:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21ccff9700 1 -- 172.21.15.46:0/3074738731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21c0012d20 con 0x7f21d00f56c0 2024-01-21T08:19:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.001+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d010f110 con 0x7f21d00f56c0 2024-01-21T08:19:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.002+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d0045ad0 con 0x7f21d00f56c0 2024-01-21T08:19:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.003+0000 7f21ccff9700 1 -- 172.21.15.46:0/3074738731 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f21c0019030 con 0x7f21d00f56c0 2024-01-21T08:19:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.003+0000 7f21ccff9700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f21b805c2a0 0x7f21b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.003+0000 7f21ccff9700 1 -- 172.21.15.46:0/3074738731 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f21c0075db0 con 0x7f21d00f56c0 2024-01-21T08:19:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.003+0000 7f21cf7fe700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f21b805c2a0 0x7f21b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.003+0000 7f21cf7fe700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f21b805c2a0 0x7f21b805e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f21c4013580 tx=0x7f21c4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:35.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.006+0000 7f21ccff9700 1 -- 172.21.15.46:0/3074738731 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21c0048e80 con 0x7f21d00f56c0 2024-01-21T08:19:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.227+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f21d00fde50 con 0x7f21d00f56c0 2024-01-21T08:19:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.227+0000 7f21ccff9700 1 -- 172.21.15.46:0/3074738731 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f21c0017020 con 0x7f21d00f56c0 2024-01-21T08:19:35.229 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f21b805c2a0 msgr2=0x7f21b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f21b805c2a0 0x7f21b805e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f21c4013580 tx=0x7f21c4009b30).stop 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f56c0 msgr2=0x7f21d00f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f56c0 0x7f21d00f9820 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f21c000c530 tx=0x7f21c0007e10).stop 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 shutdown_connections 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f21b805c2a0 0x7f21b805e760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21d00f56c0 0x7f21d00f9820 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 --2- 172.21.15.46:0/3074738731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d00f6960 0x7f21d00f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 >> 172.21.15.46:0/3074738731 conn(0x7f21d00f0e20 msgr2=0x7f21d00603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 shutdown_connections 2024-01-21T08:19:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:35.229+0000 7f21d674a700 1 -- 172.21.15.46:0/3074738731 wait complete. 2024-01-21T08:19:35.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:34.108247+0000 mgr.smithi046.lmxnwc (mgr.14164) 476 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:35.545 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:35.228133+0000 mon.smithi046 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.46:0/3074738731' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:35.621 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:34.108247+0000 mgr.smithi046.lmxnwc (mgr.14164) 476 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:35.228133+0000 mon.smithi046 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.46:0/3074738731' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:36.108934+0000 mgr.smithi046.lmxnwc (mgr.14164) 477 : cluster [DBG] pgmap v454: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:36.108934+0000 mgr.smithi046.lmxnwc (mgr.14164) 477 : cluster [DBG] pgmap v454: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:38.622 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:38.866 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:39.625 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:38.109324+0000 mgr.smithi046.lmxnwc (mgr.14164) 478 : cluster [DBG] pgmap v455: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:39.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:38.109324+0000 mgr.smithi046.lmxnwc (mgr.14164) 478 : cluster [DBG] pgmap v455: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:39.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 -- 172.21.15.46:0/2115156300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470060810 msgr2=0x7f44700f9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:39.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2115156300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470060810 0x7f44700f9400 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f4460004660 tx=0x7f4460009b30).stop 2024-01-21T08:19:39.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 -- 172.21.15.46:0/2115156300 shutdown_connections 2024-01-21T08:19:39.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2115156300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44700f9980 0x7f44700fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:19:39.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2115156300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470060810 0x7f44700f9400 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:39.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 -- 172.21.15.46:0/2115156300 >> 172.21.15.46:0/2115156300 conn(0x7f44700f0e20 msgr2=0x7f44700f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:39.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 -- 172.21.15.46:0/2115156300 shutdown_connections 2024-01-21T08:19:39.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.816+0000 7f4475ea2700 1 -- 172.21.15.46:0/2115156300 wait complete. 2024-01-21T08:19:39.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f4475ea2700 1 Processor -- start 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f4475ea2700 1 -- start start 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f4475ea2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4470060810 0x7f44700f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f4475ea2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44700f9980 0x7f44700f5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f4475ea2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700f58f0 con 0x7f44700f9980 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f4475ea2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700f5a30 con 0x7f4470060810 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f446effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44700f9980 0x7f44700f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f446effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44700f9980 0x7f44700f5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47498/0 (socket says 172.21.15.46:47498) 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f446effd700 1 -- 172.21.15.46:0/2095686019 learned_addr learned my addr 172.21.15.46:0/2095686019 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.817+0000 7f446f7fe700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4470060810 0x7f44700f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f446effd700 1 -- 172.21.15.46:0/2095686019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4470060810 msgr2=0x7f44700f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f446effd700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4470060810 0x7f44700f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f446effd700 1 -- 172.21.15.46:0/2095686019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4460005040 con 0x7f44700f9980 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f446effd700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44700f9980 0x7f44700f5300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f44580093f0 tx=0x7f4458006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:39.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f4467fff700 1 -- 172.21.15.46:0/2095686019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4458013640 con 0x7f44700f9980 2024-01-21T08:19:39.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f4467fff700 1 -- 172.21.15.46:0/2095686019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4458013c50 con 0x7f44700f9980 2024-01-21T08:19:39.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44700f5d10 con 0x7f44700f9980 2024-01-21T08:19:39.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f4467fff700 1 -- 172.21.15.46:0/2095686019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4458012c80 con 0x7f44700f9980 2024-01-21T08:19:39.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.818+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447010f110 con 0x7f44700f9980 2024-01-21T08:19:39.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.819+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4470045ad0 con 0x7f44700f9980 2024-01-21T08:19:39.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.819+0000 7f4467fff700 1 -- 172.21.15.46:0/2095686019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f445801a060 con 0x7f44700f9980 2024-01-21T08:19:39.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.820+0000 7f4467fff700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f445c05c2a0 0x7f445c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:39.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.820+0000 7f4467fff700 1 -- 172.21.15.46:0/2095686019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4458075b70 con 0x7f44700f9980 2024-01-21T08:19:39.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.820+0000 7f446f7fe700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f445c05c2a0 0x7f445c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:39.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.820+0000 7f446f7fe700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f445c05c2a0 0x7f445c05e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4460004e30 tx=0x7f4460009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:39.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:39.824+0000 7f4467fff700 1 -- 172.21.15.46:0/2095686019 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4458048b70 con 0x7f44700f9980 2024-01-21T08:19:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.042+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4470027a00 con 0x7f44700f9980 2024-01-21T08:19:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.043+0000 7f4467fff700 1 -- 172.21.15.46:0/2095686019 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f445804c190 con 0x7f44700f9980 2024-01-21T08:19:40.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f445c05c2a0 msgr2=0x7f445c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f445c05c2a0 0x7f445c05e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4460004e30 tx=0x7f4460009b30).stop 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44700f9980 msgr2=0x7f44700f5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44700f9980 0x7f44700f5300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f44580093f0 tx=0x7f4458006ec0).stop 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 shutdown_connections 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f445c05c2a0 0x7f445c05e760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4470060810 0x7f44700f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 --2- 172.21.15.46:0/2095686019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44700f9980 0x7f44700f5300 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.044+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 >> 172.21.15.46:0/2095686019 conn(0x7f44700f0e20 msgr2=0x7f44700f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.045+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 shutdown_connections 2024-01-21T08:19:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:40.045+0000 7f4475ea2700 1 -- 172.21.15.46:0/2095686019 wait complete. 2024-01-21T08:19:40.427 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:40.044721+0000 mon.smithi046 (mon.0) 511 : audit [DBG] from='client.? 172.21.15.46:0/2095686019' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:40.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:40.044721+0000 mon.smithi046 (mon.0) 511 : audit [DBG] from='client.? 172.21.15.46:0/2095686019' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:41.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:40.109686+0000 mgr.smithi046.lmxnwc (mgr.14164) 479 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:41.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:40.109686+0000 mgr.smithi046.lmxnwc (mgr.14164) 479 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:43.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:43.673 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:43.739 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:42.110000+0000 mgr.smithi046.lmxnwc (mgr.14164) 480 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:43.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:42.110000+0000 mgr.smithi046.lmxnwc (mgr.14164) 480 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 -- 172.21.15.46:0/535927736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f7a70 msgr2=0x7efc1c0f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 --2- 172.21.15.46:0/535927736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f7a70 0x7efc1c0f7ed0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7efc10004660 tx=0x7efc10009b30).stop 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 -- 172.21.15.46:0/535927736 shutdown_connections 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 --2- 172.21.15.46:0/535927736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f7a70 0x7efc1c0f7ed0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 --2- 172.21.15.46:0/535927736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1c0f67d0 0x7efc1c0f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 -- 172.21.15.46:0/535927736 >> 172.21.15.46:0/535927736 conn(0x7efc1c0f1f50 msgr2=0x7efc1c0f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 -- 172.21.15.46:0/535927736 shutdown_connections 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.594+0000 7efc24629700 1 -- 172.21.15.46:0/535927736 wait complete. 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc24629700 1 Processor -- start 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc24629700 1 -- start start 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc24629700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f67d0 0x7efc1c1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc24629700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1c0f7a70 0x7efc1c1075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc24629700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c107b60 con 0x7efc1c0f67d0 2024-01-21T08:19:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc24629700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c107ca0 con 0x7efc1c0f7a70 2024-01-21T08:19:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc223c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f67d0 0x7efc1c1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc223c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f67d0 0x7efc1c1070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51052/0 (socket says 172.21.15.46:51052) 2024-01-21T08:19:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.595+0000 7efc223c5700 1 -- 172.21.15.46:0/1885339002 learned_addr learned my addr 172.21.15.46:0/1885339002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc21bc4700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1c0f7a70 0x7efc1c1075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc223c5700 1 -- 172.21.15.46:0/1885339002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1c0f7a70 msgr2=0x7efc1c1075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc223c5700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1c0f7a70 0x7efc1c1075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc223c5700 1 -- 172.21.15.46:0/1885339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10005040 con 0x7efc1c0f67d0 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc223c5700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f67d0 0x7efc1c1070c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7efc1800c530 tx=0x7efc18007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc0effd700 1 -- 172.21.15.46:0/1885339002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc180148f0 con 0x7efc1c0f67d0 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c108640 con 0x7efc1c0f67d0 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc0effd700 1 -- 172.21.15.46:0/1885339002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc1800f470 con 0x7efc1c0f67d0 2024-01-21T08:19:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc0effd700 1 -- 172.21.15.46:0/1885339002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1801a4d0 con 0x7efc1c0f67d0 2024-01-21T08:19:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.596+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c108c00 con 0x7efc1c0f67d0 2024-01-21T08:19:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.597+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c045ad0 con 0x7efc1c0f67d0 2024-01-21T08:19:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.597+0000 7efc0effd700 1 -- 172.21.15.46:0/1885339002 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efc18010070 con 0x7efc1c0f67d0 2024-01-21T08:19:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.598+0000 7efc0effd700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc0805c2a0 0x7efc0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.598+0000 7efc0effd700 1 -- 172.21.15.46:0/1885339002 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7efc18079ef0 con 0x7efc1c0f67d0 2024-01-21T08:19:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.598+0000 7efc21bc4700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc0805c2a0 0x7efc0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.598+0000 7efc21bc4700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc0805c2a0 0x7efc0805e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7efc10013580 tx=0x7efc10009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:44.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.602+0000 7efc0effd700 1 -- 172.21.15.46:0/1885339002 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc1804cf70 con 0x7efc1c0f67d0 2024-01-21T08:19:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.822+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc1c0ff0e0 con 0x7efc1c0f67d0 2024-01-21T08:19:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.822+0000 7efc0effd700 1 -- 172.21.15.46:0/1885339002 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7efc18050590 con 0x7efc1c0f67d0 2024-01-21T08:19:44.825 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc0805c2a0 msgr2=0x7efc0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc0805c2a0 0x7efc0805e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7efc10013580 tx=0x7efc10009b30).stop 2024-01-21T08:19:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f67d0 msgr2=0x7efc1c1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f67d0 0x7efc1c1070c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7efc1800c530 tx=0x7efc18007e10).stop 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 shutdown_connections 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc0805c2a0 0x7efc0805e760 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc1c0f67d0 0x7efc1c1070c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 --2- 172.21.15.46:0/1885339002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1c0f7a70 0x7efc1c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 >> 172.21.15.46:0/1885339002 conn(0x7efc1c0f1f50 msgr2=0x7efc1c0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 shutdown_connections 2024-01-21T08:19:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:44.824+0000 7efc24629700 1 -- 172.21.15.46:0/1885339002 wait complete. 2024-01-21T08:19:45.202 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:44.110309+0000 mgr.smithi046.lmxnwc (mgr.14164) 481 : cluster [DBG] pgmap v458: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:45.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:44.824192+0000 mon.smithi046 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.46:0/1885339002' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:44.110309+0000 mgr.smithi046.lmxnwc (mgr.14164) 481 : cluster [DBG] pgmap v458: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:45.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:44.824192+0000 mon.smithi046 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.46:0/1885339002' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:47.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:46.110810+0000 mgr.smithi046.lmxnwc (mgr.14164) 482 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:46.110810+0000 mgr.smithi046.lmxnwc (mgr.14164) 482 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:48.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.403+0000 7f89360db700 1 -- 172.21.15.46:0/636259287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 msgr2=0x7f89300f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.403+0000 7f89360db700 1 --2- 172.21.15.46:0/636259287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f89300f91f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f892c004660 tx=0x7f892c009b30).stop 2024-01-21T08:19:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.403+0000 7f89360db700 1 -- 172.21.15.46:0/636259287 shutdown_connections 2024-01-21T08:19:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.403+0000 7f89360db700 1 --2- 172.21.15.46:0/636259287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89300f9770 0x7f89300fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.403+0000 7f89360db700 1 --2- 172.21.15.46:0/636259287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f89300f91f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.403+0000 7f89360db700 1 -- 172.21.15.46:0/636259287 >> 172.21.15.46:0/636259287 conn(0x7f89300f0c70 msgr2=0x7f89300f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 -- 172.21.15.46:0/636259287 shutdown_connections 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 -- 172.21.15.46:0/636259287 wait complete. 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 Processor -- start 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 -- start start 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f8930106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89300f9770 0x7f89301073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8930107990 con 0x7f8930060870 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89360db700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8930107ad0 con 0x7f89300f9770 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89350d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f8930106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89350d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f8930106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51066/0 (socket says 172.21.15.46:51066) 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.404+0000 7f89350d9700 1 -- 172.21.15.46:0/566892436 learned_addr learned my addr 172.21.15.46:0/566892436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f89348d8700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89300f9770 0x7f89301073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f89350d9700 1 -- 172.21.15.46:0/566892436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89300f9770 msgr2=0x7f89301073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f89350d9700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89300f9770 0x7f89301073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f89350d9700 1 -- 172.21.15.46:0/566892436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c005040 con 0x7f8930060870 2024-01-21T08:19:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f89350d9700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f8930106ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f892c009830 tx=0x7f892c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f8925ffb700 1 -- 172.21.15.46:0/566892436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892c018070 con 0x7f8930060870 2024-01-21T08:19:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8930108410 con 0x7f8930060870 2024-01-21T08:19:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f8925ffb700 1 -- 172.21.15.46:0/566892436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f892c013410 con 0x7f8930060870 2024-01-21T08:19:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f8925ffb700 1 -- 172.21.15.46:0/566892436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892c01c620 con 0x7f8930060870 2024-01-21T08:19:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.405+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8930108970 con 0x7f8930060870 2024-01-21T08:19:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.406+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8930045ad0 con 0x7f8930060870 2024-01-21T08:19:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.406+0000 7f8925ffb700 1 -- 172.21.15.46:0/566892436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f892c01c8e0 con 0x7f8930060870 2024-01-21T08:19:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.407+0000 7f8925ffb700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f891c05c2b0 0x7f891c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.407+0000 7f8925ffb700 1 -- 172.21.15.46:0/566892436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f892c077a60 con 0x7f8930060870 2024-01-21T08:19:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.407+0000 7f89348d8700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f891c05c2b0 0x7f891c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.407+0000 7f89348d8700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f891c05c2b0 0x7f891c05e770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f89200041d0 tx=0x7f8920009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.411+0000 7f8925ffb700 1 -- 172.21.15.46:0/566892436 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f892c04aa00 con 0x7f8930060870 2024-01-21T08:19:49.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:48.111194+0000 mgr.smithi046.lmxnwc (mgr.14164) 483 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:49.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.633+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8930027a00 con 0x7f8930060870 2024-01-21T08:19:49.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.633+0000 7f8925ffb700 1 -- 172.21.15.46:0/566892436 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f892c04e020 con 0x7f8930060870 2024-01-21T08:19:49.636 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f891c05c2b0 msgr2=0x7f891c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f891c05c2b0 0x7f891c05e770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f89200041d0 tx=0x7f8920009040).stop 2024-01-21T08:19:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 msgr2=0x7f8930106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f8930106ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f892c009830 tx=0x7f892c009b30).stop 2024-01-21T08:19:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 shutdown_connections 2024-01-21T08:19:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f891c05c2b0 0x7f891c05e770 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8930060870 0x7f8930106ef0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 --2- 172.21.15.46:0/566892436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89300f9770 0x7f89301073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 >> 172.21.15.46:0/566892436 conn(0x7f89300f0c70 msgr2=0x7f89300f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 shutdown_connections 2024-01-21T08:19:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:49.635+0000 7f89360db700 1 -- 172.21.15.46:0/566892436 wait complete. 2024-01-21T08:19:49.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:48.111194+0000 mgr.smithi046.lmxnwc (mgr.14164) 483 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:50.029 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:49.634934+0000 mon.smithi046 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.46:0/566892436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:49.634934+0000 mon.smithi046 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.46:0/566892436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:50.111701+0000 mgr.smithi046.lmxnwc (mgr.14164) 484 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:50.111701+0000 mgr.smithi046.lmxnwc (mgr.14164) 484 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:52.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:52.112039+0000 mgr.smithi046.lmxnwc (mgr.14164) 485 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:52.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:52 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:52.112039+0000 mgr.smithi046.lmxnwc (mgr.14164) 485 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:53.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:53.276 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2621414077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 msgr2=0x7f3e540f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2621414077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e540f5ae0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f3e44004660 tx=0x7f3e44009b30).stop 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2621414077 shutdown_connections 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2621414077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e540f6960 0x7f3e540f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2621414077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e540f5ae0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2621414077 >> 172.21.15.46:0/2621414077 conn(0x7f3e540f0e20 msgr2=0x7f3e540f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2621414077 shutdown_connections 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.218+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2621414077 wait complete. 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e5aa22700 1 Processor -- start 2024-01-21T08:19:54.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e5aa22700 1 -- start start 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e5aa22700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e54109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e5aa22700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e540f6960 0x7f3e5410c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e5aa22700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54060920 con 0x7f3e540f56c0 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e5aa22700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54060a60 con 0x7f3e540f6960 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e54109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e54109df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51444/0 (socket says 172.21.15.46:51444) 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.219+0000 7f3e53fff700 1 -- 172.21.15.46:0/2128939437 learned_addr learned my addr 172.21.15.46:0/2128939437 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e537fe700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e540f6960 0x7f3e5410c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e53fff700 1 -- 172.21.15.46:0/2128939437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e540f6960 msgr2=0x7f3e5410c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e53fff700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e540f6960 0x7f3e5410c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e53fff700 1 -- 172.21.15.46:0/2128939437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e44005040 con 0x7f3e540f56c0 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e53fff700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e54109df0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3e44004e30 tx=0x7f3e44009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e50ff9700 1 -- 172.21.15.46:0/2128939437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e44018070 con 0x7f3e540f56c0 2024-01-21T08:19:54.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e50ff9700 1 -- 172.21.15.46:0/2128939437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e44013410 con 0x7f3e540f56c0 2024-01-21T08:19:54.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5410c840 con 0x7f3e540f56c0 2024-01-21T08:19:54.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e50ff9700 1 -- 172.21.15.46:0/2128939437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e4401de20 con 0x7f3e540f56c0 2024-01-21T08:19:54.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.220+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5410cd70 con 0x7f3e540f56c0 2024-01-21T08:19:54.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.221+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e54059100 con 0x7f3e540f56c0 2024-01-21T08:19:54.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.222+0000 7f3e50ff9700 1 -- 172.21.15.46:0/2128939437 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3e4401d420 con 0x7f3e540f56c0 2024-01-21T08:19:54.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.222+0000 7f3e50ff9700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3e3c05c2a0 0x7f3e3c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:54.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.222+0000 7f3e50ff9700 1 -- 172.21.15.46:0/2128939437 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3e440777b0 con 0x7f3e540f56c0 2024-01-21T08:19:54.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.222+0000 7f3e537fe700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3e3c05c2a0 0x7f3e3c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:54.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.223+0000 7f3e537fe700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3e3c05c2a0 0x7f3e3c05e760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3e48004c40 tx=0x7f3e4800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:54.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.226+0000 7f3e50ff9700 1 -- 172.21.15.46:0/2128939437 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e4404a7b0 con 0x7f3e540f56c0 2024-01-21T08:19:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.448+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3e54045ad0 con 0x7f3e540f56c0 2024-01-21T08:19:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.448+0000 7f3e50ff9700 1 -- 172.21.15.46:0/2128939437 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f3e4404ddd0 con 0x7f3e540f56c0 2024-01-21T08:19:54.451 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3e3c05c2a0 msgr2=0x7f3e3c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3e3c05c2a0 0x7f3e3c05e760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3e48004c40 tx=0x7f3e4800b040).stop 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 msgr2=0x7f3e54109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e54109df0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3e44004e30 tx=0x7f3e44009b30).stop 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 shutdown_connections 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3e3c05c2a0 0x7f3e3c05e760 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e540f56c0 0x7f3e54109df0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 --2- 172.21.15.46:0/2128939437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e540f6960 0x7f3e5410c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 >> 172.21.15.46:0/2128939437 conn(0x7f3e540f0e20 msgr2=0x7f3e540fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 shutdown_connections 2024-01-21T08:19:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:54.450+0000 7f3e5aa22700 1 -- 172.21.15.46:0/2128939437 wait complete. 2024-01-21T08:19:54.861 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:55.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:54.112533+0000 mgr.smithi046.lmxnwc (mgr.14164) 486 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:55.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:54.450225+0000 mon.smithi046 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.46:0/2128939437' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:55.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:54.508596+0000 mon.smithi046 (mon.0) 515 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T08:19:55.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:54.509684+0000 mon.smithi046 (mon.0) 516 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:19:55.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:55.333427+0000 mon.smithi046 (mon.0) 517 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:19:55.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:55.334189+0000 mon.smithi046 (mon.0) 518 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi078.kcfjgo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-21T08:19:55.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:55.334898+0000 mon.smithi046 (mon.0) 519 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-21T08:19:55.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:55.335546+0000 mon.smithi046 (mon.0) 520 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:19:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:54.112533+0000 mgr.smithi046.lmxnwc (mgr.14164) 486 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:54.450225+0000 mon.smithi046 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.46:0/2128939437' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:55.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:54.508596+0000 mon.smithi046 (mon.0) 515 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi078", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T08:19:55.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:54.509684+0000 mon.smithi046 (mon.0) 516 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:19:55.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:55.333427+0000 mon.smithi046 (mon.0) 517 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:19:55.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:55.334189+0000 mon.smithi046 (mon.0) 518 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi078.kcfjgo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-21T08:19:55.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:55.334898+0000 mon.smithi046 (mon.0) 519 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-21T08:19:55.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:55.335546+0000 mon.smithi046 (mon.0) 520 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:19:56.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:54.508012+0000 mgr.smithi046.lmxnwc (mgr.14164) 487 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi046 ... 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service failed because a timeout was exceeded. 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service" and "journalctl -xe" for details. 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:19:56.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: yield (conn, connr) 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: code, '\n'.join(err))) 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi046 ... 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service failed because a timeout was exceeded. 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service" and "journalctl -xe" for details. 2024-01-21T08:19:56.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:54.508274+0000 mgr.smithi046.lmxnwc (mgr.14164) 488 : cephadm [INF] Reconfiguring crash.smithi078 (monmap changed)... 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:54.510277+0000 mgr.smithi046.lmxnwc (mgr.14164) 489 : cephadm [INF] Reconfiguring daemon crash.smithi078 on smithi078 2024-01-21T08:19:56.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:55.333929+0000 mgr.smithi046.lmxnwc (mgr.14164) 490 : cephadm [INF] Reconfiguring mgr.smithi078.kcfjgo (monmap changed)... 2024-01-21T08:19:56.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:55.336093+0000 mgr.smithi046.lmxnwc (mgr.14164) 491 : cephadm [INF] Reconfiguring daemon mgr.smithi078.kcfjgo on smithi078 2024-01-21T08:19:56.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:56.147990+0000 mon.smithi046 (mon.0) 521 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:19:56.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:56.148895+0000 mon.smithi046 (mon.0) 522 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-21T08:19:56.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:56.149679+0000 mon.smithi046 (mon.0) 523 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-21T08:19:56.744 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:56.150456+0000 mon.smithi046 (mon.0) 524 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:54.508012+0000 mgr.smithi046.lmxnwc (mgr.14164) 487 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi046 ... 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service failed because a timeout was exceeded. 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service" and "journalctl -xe" for details. 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:19:56.745 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ports=daemon_ports) 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: yield (conn, connr) 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: code, '\n'.join(err))) 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi046 ... 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service failed because a timeout was exceeded. 2024-01-21T08:19:56.746 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046.service" and "journalctl -xe" for details. 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ports=daemon_ports) 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi046 2024-01-21T08:19:56.747 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:54.508274+0000 mgr.smithi046.lmxnwc (mgr.14164) 488 : cephadm [INF] Reconfiguring crash.smithi078 (monmap changed)... 2024-01-21T08:19:56.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:54.510277+0000 mgr.smithi046.lmxnwc (mgr.14164) 489 : cephadm [INF] Reconfiguring daemon crash.smithi078 on smithi078 2024-01-21T08:19:56.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:55.333929+0000 mgr.smithi046.lmxnwc (mgr.14164) 490 : cephadm [INF] Reconfiguring mgr.smithi078.kcfjgo (monmap changed)... 2024-01-21T08:19:56.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:55.336093+0000 mgr.smithi046.lmxnwc (mgr.14164) 491 : cephadm [INF] Reconfiguring daemon mgr.smithi078.kcfjgo on smithi078 2024-01-21T08:19:56.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:56.147990+0000 mon.smithi046 (mon.0) 521 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:19:56.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:56.148895+0000 mon.smithi046 (mon.0) 522 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-21T08:19:56.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:56.149679+0000 mon.smithi046 (mon.0) 523 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-21T08:19:56.748 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:56.150456+0000 mon.smithi046 (mon.0) 524 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T08:19:57.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:56.112952+0000 mgr.smithi046.lmxnwc (mgr.14164) 492 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:57.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:56.148623+0000 mgr.smithi046.lmxnwc (mgr.14164) 493 : cephadm [INF] Reconfiguring mon.smithi078 (monmap changed)... 2024-01-21T08:19:57.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:56.151134+0000 mgr.smithi046.lmxnwc (mgr.14164) 494 : cephadm [INF] Reconfiguring daemon mon.smithi078 on smithi078 2024-01-21T08:19:57.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:56.955760+0000 mon.smithi046 (mon.0) 525 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:19:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:56.112952+0000 mgr.smithi046.lmxnwc (mgr.14164) 492 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:56.148623+0000 mgr.smithi046.lmxnwc (mgr.14164) 493 : cephadm [INF] Reconfiguring mon.smithi078 (monmap changed)... 2024-01-21T08:19:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:56.151134+0000 mgr.smithi046.lmxnwc (mgr.14164) 494 : cephadm [INF] Reconfiguring daemon mon.smithi078 on smithi078 2024-01-21T08:19:57.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:56.955760+0000 mon.smithi046 (mon.0) 525 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:19:57.862 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:19:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:19:58.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:56.956380+0000 mgr.smithi046.lmxnwc (mgr.14164) 495 : cephadm [INF] Reconfiguring node-exporter.smithi078 (unknown last config time)... 2024-01-21T08:19:58.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:19:56.956530+0000 mgr.smithi046.lmxnwc (mgr.14164) 496 : cephadm [INF] Reconfiguring daemon node-exporter.smithi078 on smithi078 2024-01-21T08:19:58.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:56.956380+0000 mgr.smithi046.lmxnwc (mgr.14164) 495 : cephadm [INF] Reconfiguring node-exporter.smithi078 (unknown last config time)... 2024-01-21T08:19:58.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:19:56.956530+0000 mgr.smithi046.lmxnwc (mgr.14164) 496 : cephadm [INF] Reconfiguring daemon node-exporter.smithi078 on smithi078 2024-01-21T08:19:59.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 -- 172.21.15.46:0/3522055426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 msgr2=0x7faba40f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 --2- 172.21.15.46:0/3522055426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba40f6dc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7faba0004660 tx=0x7faba0009b30).stop 2024-01-21T08:19:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 -- 172.21.15.46:0/3522055426 shutdown_connections 2024-01-21T08:19:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 --2- 172.21.15.46:0/3522055426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba40f6dc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 --2- 172.21.15.46:0/3522055426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faba40f56c0 0x7faba40f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 -- 172.21.15.46:0/3522055426 >> 172.21.15.46:0/3522055426 conn(0x7faba40f0e20 msgr2=0x7faba40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 -- 172.21.15.46:0/3522055426 shutdown_connections 2024-01-21T08:19:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 -- 172.21.15.46:0/3522055426 wait complete. 2024-01-21T08:19:59.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.068+0000 7fabac710700 1 Processor -- start 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabac710700 1 -- start start 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabac710700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faba40f56c0 0x7faba41070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabac710700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba41075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabac710700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4107b70 con 0x7faba40f6960 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabac710700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4107cb0 con 0x7faba40f56c0 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7faba9cab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba41075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7faba9cab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba41075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51454/0 (socket says 172.21.15.46:51454) 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7faba9cab700 1 -- 172.21.15.46:0/3521522005 learned_addr learned my addr 172.21.15.46:0/3521522005 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabaa4ac700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faba40f56c0 0x7faba41070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7faba9cab700 1 -- 172.21.15.46:0/3521522005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faba40f56c0 msgr2=0x7faba41070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7faba9cab700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faba40f56c0 0x7faba41070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7faba9cab700 1 -- 172.21.15.46:0/3521522005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba0005040 con 0x7faba40f6960 2024-01-21T08:19:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabaa4ac700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faba40f56c0 0x7faba41070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-21T08:19:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7faba9cab700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba41075c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7faba0009830 tx=0x7faba0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fab9affd700 1 -- 172.21.15.46:0/3521522005 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faba0018070 con 0x7faba40f6960 2024-01-21T08:19:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fab9affd700 1 -- 172.21.15.46:0/3521522005 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faba0013410 con 0x7faba40f6960 2024-01-21T08:19:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba41085f0 con 0x7faba40f6960 2024-01-21T08:19:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.069+0000 7fab9affd700 1 -- 172.21.15.46:0/3521522005 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faba001c620 con 0x7faba40f6960 2024-01-21T08:19:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.070+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba4108b50 con 0x7faba40f6960 2024-01-21T08:19:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.071+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba4045ad0 con 0x7faba40f6960 2024-01-21T08:19:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.071+0000 7fab9affd700 1 -- 172.21.15.46:0/3521522005 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faba001c890 con 0x7faba40f6960 2024-01-21T08:19:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.071+0000 7fab9affd700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab9005c2a0 0x7fab9005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:19:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.071+0000 7fab9affd700 1 -- 172.21.15.46:0/3521522005 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7faba0077970 con 0x7faba40f6960 2024-01-21T08:19:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.071+0000 7fabaa4ac700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab9005c2a0 0x7fab9005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:19:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.072+0000 7fabaa4ac700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab9005c2a0 0x7fab9005e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7faba40f67c0 tx=0x7fab94009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:19:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.075+0000 7fab9affd700 1 -- 172.21.15.46:0/3521522005 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faba004a990 con 0x7faba40f6960 2024-01-21T08:19:59.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.297+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faba40fdf00 con 0x7faba40f6960 2024-01-21T08:19:59.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.297+0000 7fab9affd700 1 -- 172.21.15.46:0/3521522005 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7faba004dfb0 con 0x7faba40f6960 2024-01-21T08:19:59.300 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab9005c2a0 msgr2=0x7fab9005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab9005c2a0 0x7fab9005e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7faba40f67c0 tx=0x7fab94009040).stop 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 msgr2=0x7faba41075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba41075c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7faba0009830 tx=0x7faba0009b30).stop 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 shutdown_connections 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fab9005c2a0 0x7fab9005e760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faba40f56c0 0x7faba41070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 --2- 172.21.15.46:0/3521522005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faba40f6960 0x7faba41075c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 >> 172.21.15.46:0/3521522005 conn(0x7faba40f0e20 msgr2=0x7faba40603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 shutdown_connections 2024-01-21T08:19:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:19:59.299+0000 7fabac710700 1 -- 172.21.15.46:0/3521522005 wait complete. 2024-01-21T08:19:59.600 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:19:58.113322+0000 mgr.smithi046.lmxnwc (mgr.14164) 497 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:59.600 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:19:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:19:59.299416+0000 mon.smithi046 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.46:0/3521522005' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:19:59.675 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:19:59.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:19:58.113322+0000 mgr.smithi046.lmxnwc (mgr.14164) 497 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:19:59.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:19:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:19:59.299416+0000 mon.smithi046 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.46:0/3521522005' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:00.000146+0000 mon.smithi046 (mon.0) 527 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-01-21T08:20:00.742 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:00.000167+0000 mon.smithi046 (mon.0) 528 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-01-21T08:20:00.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:00.000174+0000 mon.smithi046 (mon.0) 529 : cluster [WRN] daemon node-exporter.smithi046 on smithi046 is in unknown state 2024-01-21T08:20:00.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:00.000181+0000 mon.smithi046 (mon.0) 530 : cluster [WRN] daemon node-exporter.smithi078 on smithi078 is in unknown state 2024-01-21T08:20:00.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:00.000146+0000 mon.smithi046 (mon.0) 527 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-01-21T08:20:00.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:00.000167+0000 mon.smithi046 (mon.0) 528 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-01-21T08:20:00.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:00.000174+0000 mon.smithi046 (mon.0) 529 : cluster [WRN] daemon node-exporter.smithi046 on smithi046 is in unknown state 2024-01-21T08:20:00.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:00.000181+0000 mon.smithi046 (mon.0) 530 : cluster [WRN] daemon node-exporter.smithi078 on smithi078 is in unknown state 2024-01-21T08:20:01.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:00.113881+0000 mgr.smithi046.lmxnwc (mgr.14164) 498 : cluster [DBG] pgmap v466: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:01.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:00.113881+0000 mgr.smithi046.lmxnwc (mgr.14164) 498 : cluster [DBG] pgmap v466: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:02.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:02.708 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:02.114237+0000 mgr.smithi046.lmxnwc (mgr.14164) 499 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:02.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:02 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:02.114237+0000 mgr.smithi046.lmxnwc (mgr.14164) 499 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:02.925 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:03.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/1218676467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 msgr2=0x7fbd280f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:03.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/1218676467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f6dc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fbd18004660 tx=0x7fbd18009b30).stop 2024-01-21T08:20:03.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/1218676467 shutdown_connections 2024-01-21T08:20:03.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/1218676467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f6dc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:03.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/1218676467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd280f56c0 0x7fbd280f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:03.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/1218676467 >> 172.21.15.46:0/1218676467 conn(0x7fbd280f0e20 msgr2=0x7fbd280f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:03.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/1218676467 shutdown_connections 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.858+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/1218676467 wait complete. 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2f8b9700 1 Processor -- start 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2f8b9700 1 -- start start 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2f8b9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd280f56c0 0x7fbd280f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2f8b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2f8b9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280f8200 con 0x7fbd280f6960 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2f8b9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280f8340 con 0x7fbd280f56c0 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2ce54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2ce54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58548/0 (socket says 172.21.15.46:58548) 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2ce54700 1 -- 172.21.15.46:0/4124382513 learned_addr learned my addr 172.21.15.46:0/4124382513 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:03.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.859+0000 7fbd2d655700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd280f56c0 0x7fbd280f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd2ce54700 1 -- 172.21.15.46:0/4124382513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd280f56c0 msgr2=0x7fbd280f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd2ce54700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd280f56c0 0x7fbd280f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd2ce54700 1 -- 172.21.15.46:0/4124382513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18005040 con 0x7fbd280f6960 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd2ce54700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f7ca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fbd18009830 tx=0x7fbd18009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd1dffb700 1 -- 172.21.15.46:0/4124382513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd18018070 con 0x7fbd280f6960 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd1dffb700 1 -- 172.21.15.46:0/4124382513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd18013410 con 0x7fbd280f6960 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd280f85c0 con 0x7fbd280f6960 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd1dffb700 1 -- 172.21.15.46:0/4124382513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1801c400 con 0x7fbd280f6960 2024-01-21T08:20:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.860+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd280f8b20 con 0x7fbd280f6960 2024-01-21T08:20:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.861+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd28045ad0 con 0x7fbd280f6960 2024-01-21T08:20:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.862+0000 7fbd1dffb700 1 -- 172.21.15.46:0/4124382513 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbd1801d420 con 0x7fbd280f6960 2024-01-21T08:20:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.862+0000 7fbd1dffb700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1405c2a0 0x7fbd1405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.862+0000 7fbd1dffb700 1 -- 172.21.15.46:0/4124382513 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbd18077e30 con 0x7fbd280f6960 2024-01-21T08:20:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.862+0000 7fbd2d655700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1405c2a0 0x7fbd1405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.862+0000 7fbd2d655700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1405c2a0 0x7fbd1405e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbd280f95b0 tx=0x7fbd24009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:03.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:03.865+0000 7fbd1dffb700 1 -- 172.21.15.46:0/4124382513 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd1804aec0 con 0x7fbd280f6960 2024-01-21T08:20:04.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.085+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd280fde50 con 0x7fbd280f6960 2024-01-21T08:20:04.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.085+0000 7fbd1dffb700 1 -- 172.21.15.46:0/4124382513 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fbd1804e4e0 con 0x7fbd280f6960 2024-01-21T08:20:04.088 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:04.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1405c2a0 msgr2=0x7fbd1405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:04.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1405c2a0 0x7fbd1405e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbd280f95b0 tx=0x7fbd24009040).stop 2024-01-21T08:20:04.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 msgr2=0x7fbd280f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:04.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f7ca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fbd18009830 tx=0x7fbd18009b30).stop 2024-01-21T08:20:04.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 shutdown_connections 2024-01-21T08:20:04.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1405c2a0 0x7fbd1405e760 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:04.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd280f56c0 0x7fbd280f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:04.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 --2- 172.21.15.46:0/4124382513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd280f6960 0x7fbd280f7ca0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:04.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 >> 172.21.15.46:0/4124382513 conn(0x7fbd280f0e20 msgr2=0x7fbd280603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:04.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 shutdown_connections 2024-01-21T08:20:04.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:04.087+0000 7fbd2f8b9700 1 -- 172.21.15.46:0/4124382513 wait complete. 2024-01-21T08:20:04.391 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:04 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:04.087001+0000 mon.smithi046 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.46:0/4124382513' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:04.467 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:04.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:04.087001+0000 mon.smithi046 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.46:0/4124382513' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:04.114648+0000 mgr.smithi046.lmxnwc (mgr.14164) 500 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:04.114648+0000 mgr.smithi046.lmxnwc (mgr.14164) 500 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:07.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:07.716 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:07.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:06.114999+0000 mgr.smithi046.lmxnwc (mgr.14164) 501 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:07.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:06.114999+0000 mgr.smithi046.lmxnwc (mgr.14164) 501 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 -- 172.21.15.46:0/1923245321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 msgr2=0x7ff56c0f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 --2- 172.21.15.46:0/1923245321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c0f5ae0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff560004660 tx=0x7ff560009b30).stop 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 -- 172.21.15.46:0/1923245321 shutdown_connections 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 --2- 172.21.15.46:0/1923245321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff56c0f6960 0x7ff56c0f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 --2- 172.21.15.46:0/1923245321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c0f5ae0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 -- 172.21.15.46:0/1923245321 >> 172.21.15.46:0/1923245321 conn(0x7ff56c0f0e20 msgr2=0x7ff56c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 -- 172.21.15.46:0/1923245321 shutdown_connections 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 -- 172.21.15.46:0/1923245321 wait complete. 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 Processor -- start 2024-01-21T08:20:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.708+0000 7ff570b39700 1 -- start start 2024-01-21T08:20:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff570b39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff570b39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff56c0f6960 0x7ff56c10c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff570b39700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c060920 con 0x7ff56c0f56c0 2024-01-21T08:20:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff570b39700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c060a60 con 0x7ff56c0f6960 2024-01-21T08:20:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff56a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff56a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c109df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58564/0 (socket says 172.21.15.46:58564) 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff56a59c700 1 -- 172.21.15.46:0/3985344382 learned_addr learned my addr 172.21.15.46:0/3985344382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff569d9b700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff56c0f6960 0x7ff56c10c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff56a59c700 1 -- 172.21.15.46:0/3985344382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff56c0f6960 msgr2=0x7ff56c10c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff56a59c700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff56c0f6960 0x7ff56c10c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff56a59c700 1 -- 172.21.15.46:0/3985344382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560005040 con 0x7ff56c0f56c0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff56a59c700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c109df0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff560004e30 tx=0x7ff560009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff55affd700 1 -- 172.21.15.46:0/3985344382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff560018070 con 0x7ff56c0f56c0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff55affd700 1 -- 172.21.15.46:0/3985344382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff560013410 con 0x7ff56c0f56c0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c10c840 con 0x7ff56c0f56c0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff55affd700 1 -- 172.21.15.46:0/3985344382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff56001de20 con 0x7ff56c0f56c0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.709+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c10cd70 con 0x7ff56c0f56c0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.710+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c045ad0 con 0x7ff56c0f56c0 2024-01-21T08:20:08.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.711+0000 7ff55affd700 1 -- 172.21.15.46:0/3985344382 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff56001d420 con 0x7ff56c0f56c0 2024-01-21T08:20:08.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.711+0000 7ff55affd700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff55405c2a0 0x7ff55405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:08.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.711+0000 7ff55affd700 1 -- 172.21.15.46:0/3985344382 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff5600776e0 con 0x7ff56c0f56c0 2024-01-21T08:20:08.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.711+0000 7ff569d9b700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff55405c2a0 0x7ff55405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:08.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.712+0000 7ff569d9b700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff55405c2a0 0x7ff55405e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7ff55c004c40 tx=0x7ff55c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.715+0000 7ff55affd700 1 -- 172.21.15.46:0/3985344382 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff56004a760 con 0x7ff56c0f56c0 2024-01-21T08:20:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.933+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff56c0fdfd0 con 0x7ff56c0f56c0 2024-01-21T08:20:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.933+0000 7ff55affd700 1 -- 172.21.15.46:0/3985344382 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7ff56004dd80 con 0x7ff56c0f56c0 2024-01-21T08:20:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff55405c2a0 msgr2=0x7ff55405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff55405c2a0 0x7ff55405e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7ff55c004c40 tx=0x7ff55c00b040).stop 2024-01-21T08:20:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 msgr2=0x7ff56c109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c109df0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff560004e30 tx=0x7ff560009b30).stop 2024-01-21T08:20:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 shutdown_connections 2024-01-21T08:20:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7ff55405c2a0 0x7ff55405e760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c0f56c0 0x7ff56c109df0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 --2- 172.21.15.46:0/3985344382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff56c0f6960 0x7ff56c10c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 >> 172.21.15.46:0/3985344382 conn(0x7ff56c0f0e20 msgr2=0x7ff56c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 shutdown_connections 2024-01-21T08:20:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:08.935+0000 7ff570b39700 1 -- 172.21.15.46:0/3985344382 wait complete. 2024-01-21T08:20:09.345 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:09.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:08.115287+0000 mgr.smithi046.lmxnwc (mgr.14164) 502 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:09.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:08.935117+0000 mon.smithi046 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.46:0/3985344382' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:09.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:08.115287+0000 mgr.smithi046.lmxnwc (mgr.14164) 502 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:09.744 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:08.935117+0000 mon.smithi046 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.46:0/3985344382' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:10.115958+0000 mgr.smithi046.lmxnwc (mgr.14164) 503 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:10.115958+0000 mgr.smithi046.lmxnwc (mgr.14164) 503 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:12.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.611+0000 7f29470e9700 1 -- 172.21.15.46:0/1876729686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f67b0 msgr2=0x7f29400f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.611+0000 7f29470e9700 1 --2- 172.21.15.46:0/1876729686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f67b0 0x7f29400f6c10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f2930004660 tx=0x7f2930009b30).stop 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 -- 172.21.15.46:0/1876729686 shutdown_connections 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 --2- 172.21.15.46:0/1876729686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f67b0 0x7f29400f6c10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 --2- 172.21.15.46:0/1876729686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400f5510 0x7f29400f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 -- 172.21.15.46:0/1876729686 >> 172.21.15.46:0/1876729686 conn(0x7f29400f0c70 msgr2=0x7f29400f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 -- 172.21.15.46:0/1876729686 shutdown_connections 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 -- 172.21.15.46:0/1876729686 wait complete. 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 Processor -- start 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 -- start start 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f5510 0x7f29400f9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400f67b0 0x7f29400f8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400f8870 con 0x7f29400f5510 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.612+0000 7f29470e9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400f89b0 con 0x7f29400f67b0 2024-01-21T08:20:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29460e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f5510 0x7f29400f9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29460e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f5510 0x7f29400f9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34502/0 (socket says 172.21.15.46:34502) 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29460e7700 1 -- 172.21.15.46:0/3332471356 learned_addr learned my addr 172.21.15.46:0/3332471356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29458e6700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400f67b0 0x7f29400f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29460e7700 1 -- 172.21.15.46:0/3332471356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400f67b0 msgr2=0x7f29400f8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29460e7700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400f67b0 0x7f29400f8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29460e7700 1 -- 172.21.15.46:0/3332471356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930005040 con 0x7f29400f5510 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29460e7700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f5510 0x7f29400f9e90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f293c00c790 tx=0x7f293c007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f2936ffd700 1 -- 172.21.15.46:0/3332471356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f293c00e680 con 0x7f29400f5510 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f2936ffd700 1 -- 172.21.15.46:0/3332471356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f293c00ec90 con 0x7f29400f5510 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29400f8c90 con 0x7f29400f5510 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f2936ffd700 1 -- 172.21.15.46:0/3332471356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f293c012cb0 con 0x7f29400f5510 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.613+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294010ef10 con 0x7f29400f5510 2024-01-21T08:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.614+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2940045ad0 con 0x7f29400f5510 2024-01-21T08:20:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.615+0000 7f2936ffd700 1 -- 172.21.15.46:0/3332471356 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f293c010070 con 0x7f29400f5510 2024-01-21T08:20:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.615+0000 7f2936ffd700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f292c05c2b0 0x7f292c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.615+0000 7f2936ffd700 1 -- 172.21.15.46:0/3332471356 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f293c075b80 con 0x7f29400f5510 2024-01-21T08:20:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.615+0000 7f29458e6700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f292c05c2b0 0x7f292c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.615+0000 7f29458e6700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f292c05c2b0 0x7f292c05e770 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f2930009d60 tx=0x7f2930009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.619+0000 7f2936ffd700 1 -- 172.21.15.46:0/3332471356 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f293c048ba0 con 0x7f29400f5510 2024-01-21T08:20:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:12.116314+0000 mgr.smithi046.lmxnwc (mgr.14164) 504 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:12.116314+0000 mgr.smithi046.lmxnwc (mgr.14164) 504 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:13.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.841+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f294010f620 con 0x7f29400f5510 2024-01-21T08:20:13.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.842+0000 7f2936ffd700 1 -- 172.21.15.46:0/3332471356 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f293c04c1c0 con 0x7f29400f5510 2024-01-21T08:20:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f292c05c2b0 msgr2=0x7f292c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f292c05c2b0 0x7f292c05e770 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f2930009d60 tx=0x7f2930009b30).stop 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f5510 msgr2=0x7f29400f9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f5510 0x7f29400f9e90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f293c00c790 tx=0x7f293c007d90).stop 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 shutdown_connections 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f292c05c2b0 0x7f292c05e770 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400f5510 0x7f29400f9e90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 --2- 172.21.15.46:0/3332471356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400f67b0 0x7f29400f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 >> 172.21.15.46:0/3332471356 conn(0x7f29400f0c70 msgr2=0x7f294005fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:13.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 shutdown_connections 2024-01-21T08:20:13.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:13.843+0000 7f29470e9700 1 -- 172.21.15.46:0/3332471356 wait complete. 2024-01-21T08:20:14.224 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:14.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:13.842522+0000 mon.smithi046 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.46:0/3332471356' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:14.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:14.116795+0000 mgr.smithi046.lmxnwc (mgr.14164) 505 : cluster [DBG] pgmap v473: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:14.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:14 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:13.842522+0000 mon.smithi046 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.46:0/3332471356' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:14.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:14 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:14.116795+0000 mgr.smithi046.lmxnwc (mgr.14164) 505 : cluster [DBG] pgmap v473: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:17.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:17.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:16.117281+0000 mgr.smithi046.lmxnwc (mgr.14164) 506 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:17.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:16.117281+0000 mgr.smithi046.lmxnwc (mgr.14164) 506 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.423+0000 7fc5fa95b700 1 -- 172.21.15.46:0/1194976864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f6610 msgr2=0x7fc5f40f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.423+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/1194976864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f6610 0x7fc5f40f6a30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fc5f0004660 tx=0x7fc5f0009b30).stop 2024-01-21T08:20:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.423+0000 7fc5fa95b700 1 -- 172.21.15.46:0/1194976864 shutdown_connections 2024-01-21T08:20:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.423+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/1194976864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f40f78b0 0x7fc5f40f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.423+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/1194976864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f6610 0x7fc5f40f6a30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 -- 172.21.15.46:0/1194976864 >> 172.21.15.46:0/1194976864 conn(0x7fc5f40f1d70 msgr2=0x7fc5f40f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 -- 172.21.15.46:0/1194976864 shutdown_connections 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 -- 172.21.15.46:0/1194976864 wait complete. 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 Processor -- start 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 -- start start 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f40f6610 0x7fc5f4106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f78b0 0x7fc5f41073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4107980 con 0x7fc5f40f78b0 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.424+0000 7fc5fa95b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4107ac0 con 0x7fc5f40f6610 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9158700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f78b0 0x7fc5f41073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9158700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f78b0 0x7fc5f41073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34514/0 (socket says 172.21.15.46:34514) 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9158700 1 -- 172.21.15.46:0/3302577541 learned_addr learned my addr 172.21.15.46:0/3302577541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9959700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f40f6610 0x7fc5f4106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9158700 1 -- 172.21.15.46:0/3302577541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f40f6610 msgr2=0x7fc5f4106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:18.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9158700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f40f6610 0x7fc5f4106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:18.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9158700 1 -- 172.21.15.46:0/3302577541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f0005040 con 0x7fc5f40f78b0 2024-01-21T08:20:18.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5f9158700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f78b0 0x7fc5f41073d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc5e40026d0 tx=0x7fc5e4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:18.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5ea7fc700 1 -- 172.21.15.46:0/3302577541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e40079c0 con 0x7fc5f40f78b0 2024-01-21T08:20:18.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5ea7fc700 1 -- 172.21.15.46:0/3302577541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5e400f040 con 0x7fc5f40f78b0 2024-01-21T08:20:18.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f4108460 con 0x7fc5f40f78b0 2024-01-21T08:20:18.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5ea7fc700 1 -- 172.21.15.46:0/3302577541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e40148b0 con 0x7fc5f40f78b0 2024-01-21T08:20:18.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.425+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f4108a20 con 0x7fc5f40f78b0 2024-01-21T08:20:18.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.426+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f4045ad0 con 0x7fc5f40f78b0 2024-01-21T08:20:18.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.427+0000 7fc5ea7fc700 1 -- 172.21.15.46:0/3302577541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc5e4007b20 con 0x7fc5f40f78b0 2024-01-21T08:20:18.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.427+0000 7fc5ea7fc700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc5e005c2b0 0x7fc5e005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:18.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.427+0000 7fc5ea7fc700 1 -- 172.21.15.46:0/3302577541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc5e4075b00 con 0x7fc5f40f78b0 2024-01-21T08:20:18.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.427+0000 7fc5f9959700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc5e005c2b0 0x7fc5e005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:18.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.427+0000 7fc5f9959700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc5e005c2b0 0x7fc5e005e770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fc5f0009f50 tx=0x7fc5f0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:18.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.431+0000 7fc5ea7fc700 1 -- 172.21.15.46:0/3302577541 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5e4048c30 con 0x7fc5f40f78b0 2024-01-21T08:20:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.649+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc5f4027a00 con 0x7fc5f40f78b0 2024-01-21T08:20:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.649+0000 7fc5ea7fc700 1 -- 172.21.15.46:0/3302577541 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fc5e404c250 con 0x7fc5f40f78b0 2024-01-21T08:20:18.651 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc5e005c2b0 msgr2=0x7fc5e005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc5e005c2b0 0x7fc5e005e770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fc5f0009f50 tx=0x7fc5f0009b30).stop 2024-01-21T08:20:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f78b0 msgr2=0x7fc5f41073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f78b0 0x7fc5f41073d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc5e40026d0 tx=0x7fc5e4009de0).stop 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 shutdown_connections 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc5e005c2b0 0x7fc5e005e770 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f40f6610 0x7fc5f4106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 --2- 172.21.15.46:0/3302577541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f40f78b0 0x7fc5f41073d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 >> 172.21.15.46:0/3302577541 conn(0x7fc5f40f1d70 msgr2=0x7fc5f40fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 shutdown_connections 2024-01-21T08:20:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:18.651+0000 7fc5fa95b700 1 -- 172.21.15.46:0/3302577541 wait complete. 2024-01-21T08:20:19.062 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:19.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:18.117596+0000 mgr.smithi046.lmxnwc (mgr.14164) 507 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:19.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:18.650332+0000 mon.smithi046 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.46:0/3302577541' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:19.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:18.117596+0000 mgr.smithi046.lmxnwc (mgr.14164) 507 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:19.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:18.650332+0000 mon.smithi046 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.46:0/3302577541' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:20.117963+0000 mgr.smithi046.lmxnwc (mgr.14164) 508 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:20.762259+0000 mon.smithi046 (mon.0) 535 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:20:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:20.762527+0000 mon.smithi046 (mon.0) 536 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:20:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:20.117963+0000 mgr.smithi046.lmxnwc (mgr.14164) 508 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:20.762259+0000 mon.smithi046 (mon.0) 535 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:20:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:20.762527+0000 mon.smithi046 (mon.0) 536 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:20:22.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:22.311 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 -- 172.21.15.46:0/1921290675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 msgr2=0x7f49200f9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 --2- 172.21.15.46:0/1921290675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f9400 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f4910004660 tx=0x7f4910009b30).stop 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 -- 172.21.15.46:0/1921290675 shutdown_connections 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 --2- 172.21.15.46:0/1921290675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49200f9980 0x7f49200fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 --2- 172.21.15.46:0/1921290675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f9400 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 -- 172.21.15.46:0/1921290675 >> 172.21.15.46:0/1921290675 conn(0x7f49200f0e20 msgr2=0x7f49200f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 -- 172.21.15.46:0/1921290675 shutdown_connections 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.277+0000 7f4926254700 1 -- 172.21.15.46:0/1921290675 wait complete. 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f4926254700 1 Processor -- start 2024-01-21T08:20:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f4926254700 1 -- start start 2024-01-21T08:20:23.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f4926254700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f4926254700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49200f9980 0x7f49200f5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f4926254700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49200f5860 con 0x7f4920060810 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f4926254700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49200f59a0 con 0x7f49200f9980 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f491f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f491f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52834/0 (socket says 172.21.15.46:52834) 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.278+0000 7f491f7fe700 1 -- 172.21.15.46:0/433854684 learned_addr learned my addr 172.21.15.46:0/433854684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f491effd700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49200f9980 0x7f49200f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f491f7fe700 1 -- 172.21.15.46:0/433854684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49200f9980 msgr2=0x7f49200f5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f491f7fe700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49200f9980 0x7f49200f5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f491f7fe700 1 -- 172.21.15.46:0/433854684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910005040 con 0x7f4920060810 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f491effd700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49200f9980 0x7f49200f5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f491f7fe700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f6e80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f4910009830 tx=0x7f4910009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f4917fff700 1 -- 172.21.15.46:0/433854684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4910018070 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49200f5c20 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f4917fff700 1 -- 172.21.15.46:0/433854684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4910013410 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f4917fff700 1 -- 172.21.15.46:0/433854684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f491001c620 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.279+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49200f6180 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.280+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4920045ad0 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.280+0000 7f4917fff700 1 -- 172.21.15.46:0/433854684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f491001c8e0 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.281+0000 7f4917fff700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f490c05c2a0 0x7f490c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.281+0000 7f4917fff700 1 -- 172.21.15.46:0/433854684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4910077a80 con 0x7f4920060810 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.281+0000 7f491effd700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f490c05c2a0 0x7f490c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.281+0000 7f491effd700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f490c05c2a0 0x7f490c05e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f49080041d0 tx=0x7f4908009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:23.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.285+0000 7f4917fff700 1 -- 172.21.15.46:0/433854684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f491004aa20 con 0x7f4920060810 2024-01-21T08:20:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.506+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4920027a00 con 0x7f4920060810 2024-01-21T08:20:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.507+0000 7f4917fff700 1 -- 172.21.15.46:0/433854684 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f491004e040 con 0x7f4920060810 2024-01-21T08:20:23.509 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f490c05c2a0 msgr2=0x7f490c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f490c05c2a0 0x7f490c05e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f49080041d0 tx=0x7f4908009040).stop 2024-01-21T08:20:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 msgr2=0x7f49200f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f6e80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f4910009830 tx=0x7f4910009b30).stop 2024-01-21T08:20:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 shutdown_connections 2024-01-21T08:20:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f490c05c2a0 0x7f490c05e760 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4920060810 0x7f49200f6e80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 --2- 172.21.15.46:0/433854684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49200f9980 0x7f49200f5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.508+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 >> 172.21.15.46:0/433854684 conn(0x7f49200f0e20 msgr2=0x7f49200f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.509+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 shutdown_connections 2024-01-21T08:20:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:23.509+0000 7f4926254700 1 -- 172.21.15.46:0/433854684 wait complete. 2024-01-21T08:20:23.668 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:22.118276+0000 mgr.smithi046.lmxnwc (mgr.14164) 509 : cluster [DBG] pgmap v477: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:23.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:22.118276+0000 mgr.smithi046.lmxnwc (mgr.14164) 509 : cluster [DBG] pgmap v477: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:23.890 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:24.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:23.507748+0000 mon.smithi046 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.46:0/433854684' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:24.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:23.507748+0000 mon.smithi046 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.46:0/433854684' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:24.118660+0000 mgr.smithi046.lmxnwc (mgr.14164) 510 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:24.118660+0000 mgr.smithi046.lmxnwc (mgr.14164) 510 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:26.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:26.119033+0000 mgr.smithi046.lmxnwc (mgr.14164) 511 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:26.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:26.119033+0000 mgr.smithi046.lmxnwc (mgr.14164) 511 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:26.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 -- 172.21.15.46:0/3562572611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 msgr2=0x7efd740f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 --2- 172.21.15.46:0/3562572611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd740f5ae0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7efd70004660 tx=0x7efd70009b30).stop 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 -- 172.21.15.46:0/3562572611 shutdown_connections 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 --2- 172.21.15.46:0/3562572611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd740f6960 0x7efd740f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 --2- 172.21.15.46:0/3562572611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd740f5ae0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 -- 172.21.15.46:0/3562572611 >> 172.21.15.46:0/3562572611 conn(0x7efd740f0e20 msgr2=0x7efd740f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 -- 172.21.15.46:0/3562572611 shutdown_connections 2024-01-21T08:20:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.141+0000 7efd7baf7700 1 -- 172.21.15.46:0/3562572611 wait complete. 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd7baf7700 1 Processor -- start 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd7baf7700 1 -- start start 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd7baf7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd74105fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd7baf7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd740f6960 0x7efd741064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd7baf7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74106a70 con 0x7efd740f56c0 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd7baf7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74106bb0 con 0x7efd740f6960 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd79893700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd74105fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd79893700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd74105fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52846/0 (socket says 172.21.15.46:52846) 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.142+0000 7efd79893700 1 -- 172.21.15.46:0/1048338426 learned_addr learned my addr 172.21.15.46:0/1048338426 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd79893700 1 -- 172.21.15.46:0/1048338426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd740f6960 msgr2=0x7efd741064c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd79893700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd740f6960 0x7efd741064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd79893700 1 -- 172.21.15.46:0/1048338426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd70005040 con 0x7efd740f56c0 2024-01-21T08:20:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd79893700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd74105fd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7efd70009f50 tx=0x7efd70009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd6a7fc700 1 -- 172.21.15.46:0/1048338426 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd70018070 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd6a7fc700 1 -- 172.21.15.46:0/1048338426 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd70013410 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd741074f0 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd6a7fc700 1 -- 172.21.15.46:0/1048338426 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd7001c620 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.143+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd74107a50 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.144+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd74045ad0 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.144+0000 7efd6a7fc700 1 -- 172.21.15.46:0/1048338426 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efd7001d420 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.145+0000 7efd6a7fc700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efd6005c2a0 0x7efd6005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.145+0000 7efd6a7fc700 1 -- 172.21.15.46:0/1048338426 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7efd70077950 con 0x7efd740f56c0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.145+0000 7efd79092700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efd6005c2a0 0x7efd6005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.145+0000 7efd79092700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efd6005c2a0 0x7efd6005e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7efd640047d0 tx=0x7efd64007a40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:28.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.149+0000 7efd6a7fc700 1 -- 172.21.15.46:0/1048338426 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd7004a9d0 con 0x7efd740f56c0 2024-01-21T08:20:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.368+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efd74027a00 con 0x7efd740f56c0 2024-01-21T08:20:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.369+0000 7efd6a7fc700 1 -- 172.21.15.46:0/1048338426 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7efd7004dff0 con 0x7efd740f56c0 2024-01-21T08:20:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efd6005c2a0 msgr2=0x7efd6005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efd6005c2a0 0x7efd6005e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7efd640047d0 tx=0x7efd64007a40).stop 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 msgr2=0x7efd74105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd74105fd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7efd70009f50 tx=0x7efd70009b30).stop 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 shutdown_connections 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efd6005c2a0 0x7efd6005e760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd740f56c0 0x7efd74105fd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 --2- 172.21.15.46:0/1048338426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd740f6960 0x7efd741064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 >> 172.21.15.46:0/1048338426 conn(0x7efd740f0e20 msgr2=0x7efd740fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 shutdown_connections 2024-01-21T08:20:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:28.370+0000 7efd7baf7700 1 -- 172.21.15.46:0/1048338426 wait complete. 2024-01-21T08:20:28.756 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:29.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:28.119400+0000 mgr.smithi046.lmxnwc (mgr.14164) 512 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:29.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:28.369702+0000 mon.smithi046 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.46:0/1048338426' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:29.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:28.119400+0000 mgr.smithi046.lmxnwc (mgr.14164) 512 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:29.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:28.369702+0000 mon.smithi046 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.46:0/1048338426' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:31.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:30.119797+0000 mgr.smithi046.lmxnwc (mgr.14164) 513 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:31.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:30.119797+0000 mgr.smithi046.lmxnwc (mgr.14164) 513 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:31.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:32.003 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.993+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3243001554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f67c0 msgr2=0x7fe4880f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.993+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3243001554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f67c0 0x7fe4880f6be0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fe478004660 tx=0x7fe478009b30).stop 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.993+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3243001554 shutdown_connections 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.993+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3243001554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4880f7a60 0x7fe4880f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.993+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3243001554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f67c0 0x7fe4880f6be0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3243001554 >> 172.21.15.46:0/3243001554 conn(0x7fe4880f1f20 msgr2=0x7fe4880f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3243001554 shutdown_connections 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3243001554 wait complete. 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 Processor -- start 2024-01-21T08:20:32.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 -- start start 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4880f67c0 0x7fe488107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f7a60 0x7fe488107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488107ad0 con 0x7fe4880f7a60 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe48ca1d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488107c10 con 0x7fe4880f67c0 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.994+0000 7fe485d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f7a60 0x7fe488107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe485d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f7a60 0x7fe488107520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34414/0 (socket says 172.21.15.46:34414) 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe485d9b700 1 -- 172.21.15.46:0/3244179789 learned_addr learned my addr 172.21.15.46:0/3244179789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe48659c700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4880f67c0 0x7fe488107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe485d9b700 1 -- 172.21.15.46:0/3244179789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4880f67c0 msgr2=0x7fe488107030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe485d9b700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4880f67c0 0x7fe488107030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe485d9b700 1 -- 172.21.15.46:0/3244179789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe478005040 con 0x7fe4880f7a60 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe48659c700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4880f67c0 0x7fe488107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-21T08:20:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe485d9b700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f7a60 0x7fe488107520 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fe47c00e7f0 tx=0x7fe47c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe476ffd700 1 -- 172.21.15.46:0/3244179789 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47c0092e0 con 0x7fe4880f7a60 2024-01-21T08:20:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe476ffd700 1 -- 172.21.15.46:0/3244179789 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe47c00f040 con 0x7fe4880f7a60 2024-01-21T08:20:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4881085b0 con 0x7fe4880f7a60 2024-01-21T08:20:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe476ffd700 1 -- 172.21.15.46:0/3244179789 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47c0146c0 con 0x7fe4880f7a60 2024-01-21T08:20:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.995+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe488108b70 con 0x7fe4880f7a60 2024-01-21T08:20:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.996+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe488045ad0 con 0x7fe4880f7a60 2024-01-21T08:20:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.996+0000 7fe476ffd700 1 -- 172.21.15.46:0/3244179789 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe47c01b030 con 0x7fe4880f7a60 2024-01-21T08:20:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.997+0000 7fe476ffd700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe47005c2a0 0x7fe47005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.997+0000 7fe476ffd700 1 -- 172.21.15.46:0/3244179789 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe47c075800 con 0x7fe4880f7a60 2024-01-21T08:20:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.997+0000 7fe48659c700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe47005c2a0 0x7fe47005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:32.997+0000 7fe48659c700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe47005c2a0 0x7fe47005e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe478004d80 tx=0x7fe47800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.001+0000 7fe476ffd700 1 -- 172.21.15.46:0/3244179789 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe47c048850 con 0x7fe4880f7a60 2024-01-21T08:20:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.220+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe488027a00 con 0x7fe4880f7a60 2024-01-21T08:20:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.221+0000 7fe476ffd700 1 -- 172.21.15.46:0/3244179789 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fe47c019080 con 0x7fe4880f7a60 2024-01-21T08:20:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:33.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe47005c2a0 msgr2=0x7fe47005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:33.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe47005c2a0 0x7fe47005e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe478004d80 tx=0x7fe47800b040).stop 2024-01-21T08:20:33.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f7a60 msgr2=0x7fe488107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:33.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f7a60 0x7fe488107520 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fe47c00e7f0 tx=0x7fe47c00b040).stop 2024-01-21T08:20:33.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 shutdown_connections 2024-01-21T08:20:33.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe47005c2a0 0x7fe47005e760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:33.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4880f67c0 0x7fe488107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:33.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 --2- 172.21.15.46:0/3244179789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4880f7a60 0x7fe488107520 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:33.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 >> 172.21.15.46:0/3244179789 conn(0x7fe4880f1f20 msgr2=0x7fe4880fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:33.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 shutdown_connections 2024-01-21T08:20:33.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:33.223+0000 7fe48ca1d700 1 -- 172.21.15.46:0/3244179789 wait complete. 2024-01-21T08:20:33.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:32.120044+0000 mgr.smithi046.lmxnwc (mgr.14164) 514 : cluster [DBG] pgmap v482: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:33.520 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:33.221743+0000 mon.smithi046 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.46:0/3244179789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:33.604 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:33.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:32.120044+0000 mgr.smithi046.lmxnwc (mgr.14164) 514 : cluster [DBG] pgmap v482: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:33.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:33.221743+0000 mon.smithi046 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.46:0/3244179789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:34.120439+0000 mgr.smithi046.lmxnwc (mgr.14164) 515 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:34.120439+0000 mgr.smithi046.lmxnwc (mgr.14164) 515 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:36.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:37.593 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:36.120924+0000 mgr.smithi046.lmxnwc (mgr.14164) 516 : cluster [DBG] pgmap v484: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:36.120924+0000 mgr.smithi046.lmxnwc (mgr.14164) 516 : cluster [DBG] pgmap v484: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 -- 172.21.15.46:0/2318534407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 msgr2=0x7f4e280f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 --2- 172.21.15.46:0/2318534407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e280f6c00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f4e24004660 tx=0x7f4e24009b30).stop 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 -- 172.21.15.46:0/2318534407 shutdown_connections 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 --2- 172.21.15.46:0/2318534407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e280f7a80 0x7f4e280f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 --2- 172.21.15.46:0/2318534407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e280f6c00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 -- 172.21.15.46:0/2318534407 >> 172.21.15.46:0/2318534407 conn(0x7f4e280f1f80 msgr2=0x7f4e280f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 -- 172.21.15.46:0/2318534407 shutdown_connections 2024-01-21T08:20:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.800+0000 7f4e2f978700 1 -- 172.21.15.46:0/2318534407 wait complete. 2024-01-21T08:20:37.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2f978700 1 Processor -- start 2024-01-21T08:20:37.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2f978700 1 -- start start 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2f978700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e28107020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2f978700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e280f7a80 0x7f4e28107510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2f978700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e28107ac0 con 0x7f4e280f67e0 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2f978700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e28107c00 con 0x7f4e280f7a80 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2d714700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e28107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2d714700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e28107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34424/0 (socket says 172.21.15.46:34424) 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2d714700 1 -- 172.21.15.46:0/617241504 learned_addr learned my addr 172.21.15.46:0/617241504 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2cf13700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e280f7a80 0x7f4e28107510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2d714700 1 -- 172.21.15.46:0/617241504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e280f7a80 msgr2=0x7f4e28107510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2d714700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e280f7a80 0x7f4e28107510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2d714700 1 -- 172.21.15.46:0/617241504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24005040 con 0x7f4e280f67e0 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e2d714700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e28107020 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f4e24000c00 tx=0x7f4e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.801+0000 7f4e19ffb700 1 -- 172.21.15.46:0/617241504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e24019070 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.802+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e28108540 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.802+0000 7f4e19ffb700 1 -- 172.21.15.46:0/617241504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e24007e70 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.802+0000 7f4e19ffb700 1 -- 172.21.15.46:0/617241504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e24014d40 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.802+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e28108aa0 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.803+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e28045ad0 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.803+0000 7f4e19ffb700 1 -- 172.21.15.46:0/617241504 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4e24025070 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.803+0000 7f4e19ffb700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e1405c2a0 0x7f4e1405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.803+0000 7f4e19ffb700 1 -- 172.21.15.46:0/617241504 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4e24076b10 con 0x7f4e280f67e0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.803+0000 7f4e2cf13700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e1405c2a0 0x7f4e1405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.804+0000 7f4e2cf13700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e1405c2a0 0x7f4e1405e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4e1c009fd0 tx=0x7f4e1c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:37.807+0000 7f4e19ffb700 1 -- 172.21.15.46:0/617241504 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e24049b10 con 0x7f4e280f67e0 2024-01-21T08:20:38.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.028+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4e28027a00 con 0x7f4e280f67e0 2024-01-21T08:20:38.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.028+0000 7f4e19ffb700 1 -- 172.21.15.46:0/617241504 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f4e2404d130 con 0x7f4e280f67e0 2024-01-21T08:20:38.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e1405c2a0 msgr2=0x7f4e1405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e1405c2a0 0x7f4e1405e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4e1c009fd0 tx=0x7f4e1c00b040).stop 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 msgr2=0x7f4e28107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e28107020 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f4e24000c00 tx=0x7f4e2400b040).stop 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 shutdown_connections 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e1405c2a0 0x7f4e1405e760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e280f67e0 0x7f4e28107020 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 --2- 172.21.15.46:0/617241504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e280f7a80 0x7f4e28107510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 >> 172.21.15.46:0/617241504 conn(0x7f4e280f1f80 msgr2=0x7f4e280fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 shutdown_connections 2024-01-21T08:20:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:38.030+0000 7f4e2f978700 1 -- 172.21.15.46:0/617241504 wait complete. 2024-01-21T08:20:38.387 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:38.029266+0000 mon.smithi046 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.46:0/617241504' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:38.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:38 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:38.029266+0000 mon.smithi046 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.46:0/617241504' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:39.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:38.121273+0000 mgr.smithi046.lmxnwc (mgr.14164) 517 : cluster [DBG] pgmap v485: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:39.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:38.121273+0000 mgr.smithi046.lmxnwc (mgr.14164) 517 : cluster [DBG] pgmap v485: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:40.121672+0000 mgr.smithi046.lmxnwc (mgr.14164) 518 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:40.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:40.121672+0000 mgr.smithi046.lmxnwc (mgr.14164) 518 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:41.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:41.633 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/3575846146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 msgr2=0x7fcf840f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/3575846146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf840f6dc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fcf78004660 tx=0x7fcf78009b30).stop 2024-01-21T08:20:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/3575846146 shutdown_connections 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/3575846146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf840f6dc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/3575846146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf840f56c0 0x7fcf840f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/3575846146 >> 172.21.15.46:0/3575846146 conn(0x7fcf840f0e20 msgr2=0x7fcf840f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/3575846146 shutdown_connections 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.650+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/3575846146 wait complete. 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf8ba3b700 1 Processor -- start 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf8ba3b700 1 -- start start 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf8ba3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf840f56c0 0x7fcf84109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf8ba3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf8410c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf8ba3b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf84060920 con 0x7fcf840f6960 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf8ba3b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf84060a60 con 0x7fcf840f56c0 2024-01-21T08:20:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf88fd6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf8410c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf88fd6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf8410c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37940/0 (socket says 172.21.15.46:37940) 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf88fd6700 1 -- 172.21.15.46:0/4249785470 learned_addr learned my addr 172.21.15.46:0/4249785470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.651+0000 7fcf897d7700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf840f56c0 0x7fcf84109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf88fd6700 1 -- 172.21.15.46:0/4249785470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf840f56c0 msgr2=0x7fcf84109dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf88fd6700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf840f56c0 0x7fcf84109dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf88fd6700 1 -- 172.21.15.46:0/4249785470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78005040 con 0x7fcf840f6960 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf897d7700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf840f56c0 0x7fcf84109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf88fd6700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf8410c2d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fcf78009830 tx=0x7fcf78009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf75ffb700 1 -- 172.21.15.46:0/4249785470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf78018070 con 0x7fcf840f6960 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf75ffb700 1 -- 172.21.15.46:0/4249785470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf78013410 con 0x7fcf840f6960 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8410c820 con 0x7fcf840f6960 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf75ffb700 1 -- 172.21.15.46:0/4249785470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf7801c620 con 0x7fcf840f6960 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.652+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8410cd50 con 0x7fcf840f6960 2024-01-21T08:20:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.653+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf84059100 con 0x7fcf840f6960 2024-01-21T08:20:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.653+0000 7fcf75ffb700 1 -- 172.21.15.46:0/4249785470 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcf7801c890 con 0x7fcf840f6960 2024-01-21T08:20:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.654+0000 7fcf75ffb700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcf7005c2a0 0x7fcf7005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.654+0000 7fcf75ffb700 1 -- 172.21.15.46:0/4249785470 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fcf780779b0 con 0x7fcf840f6960 2024-01-21T08:20:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.654+0000 7fcf897d7700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcf7005c2a0 0x7fcf7005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.654+0000 7fcf897d7700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcf7005c2a0 0x7fcf7005e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fcf840605a0 tx=0x7fcf80009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.658+0000 7fcf75ffb700 1 -- 172.21.15.46:0/4249785470 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf7804a9d0 con 0x7fcf840f6960 2024-01-21T08:20:42.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.877+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf840fdfe0 con 0x7fcf840f6960 2024-01-21T08:20:42.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.878+0000 7fcf75ffb700 1 -- 172.21.15.46:0/4249785470 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fcf840fdfe0 con 0x7fcf840f6960 2024-01-21T08:20:42.884 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.879+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcf7005c2a0 msgr2=0x7fcf7005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.879+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcf7005c2a0 0x7fcf7005e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fcf840605a0 tx=0x7fcf80009040).stop 2024-01-21T08:20:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.879+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 msgr2=0x7fcf8410c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.879+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf8410c2d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fcf78009830 tx=0x7fcf78009b30).stop 2024-01-21T08:20:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.880+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 shutdown_connections 2024-01-21T08:20:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.880+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fcf7005c2a0 0x7fcf7005e760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.880+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf840f56c0 0x7fcf84109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.880+0000 7fcf8ba3b700 1 --2- 172.21.15.46:0/4249785470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840f6960 0x7fcf8410c2d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.880+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 >> 172.21.15.46:0/4249785470 conn(0x7fcf840f0e20 msgr2=0x7fcf840fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.880+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 shutdown_connections 2024-01-21T08:20:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:42.880+0000 7fcf8ba3b700 1 -- 172.21.15.46:0/4249785470 wait complete. 2024-01-21T08:20:43.263 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:43.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:42.121939+0000 mgr.smithi046.lmxnwc (mgr.14164) 519 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:43.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:42.878849+0000 mon.smithi046 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.46:0/4249785470' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:43.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:42.121939+0000 mgr.smithi046.lmxnwc (mgr.14164) 519 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:43.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:42.878849+0000 mon.smithi046 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.46:0/4249785470' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:45.743 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:44.122259+0000 mgr.smithi046.lmxnwc (mgr.14164) 520 : cluster [DBG] pgmap v488: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:45.743 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:44.122259+0000 mgr.smithi046.lmxnwc (mgr.14164) 520 : cluster [DBG] pgmap v488: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:46.264 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:47.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:46.122767+0000 mgr.smithi046.lmxnwc (mgr.14164) 521 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 -- 172.21.15.46:0/3300468923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06880f9920 msgr2=0x7f06880fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 --2- 172.21.15.46:0/3300468923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06880f9920 0x7f06880fbd70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f067c004660 tx=0x7f067c009b30).stop 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 -- 172.21.15.46:0/3300468923 shutdown_connections 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 --2- 172.21.15.46:0/3300468923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06880f9920 0x7f06880fbd70 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 --2- 172.21.15.46:0/3300468923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0688060870 0x7f06880f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 -- 172.21.15.46:0/3300468923 >> 172.21.15.46:0/3300468923 conn(0x7f06880f0e20 msgr2=0x7f06880f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 -- 172.21.15.46:0/3300468923 shutdown_connections 2024-01-21T08:20:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 -- 172.21.15.46:0/3300468923 wait complete. 2024-01-21T08:20:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.571+0000 7f068e4f1700 1 Processor -- start 2024-01-21T08:20:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f068e4f1700 1 -- start start 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f068e4f1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688060870 0x7f06880f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f068e4f1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06880f9920 0x7f06880f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f068e4f1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880f5800 con 0x7f0688060870 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f068e4f1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880f5940 con 0x7f06880f9920 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f0687fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688060870 0x7f06880f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f0687fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688060870 0x7f06880f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37944/0 (socket says 172.21.15.46:37944) 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f0687fff700 1 -- 172.21.15.46:0/2009374742 learned_addr learned my addr 172.21.15.46:0/2009374742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f06877fe700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06880f9920 0x7f06880f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f0687fff700 1 -- 172.21.15.46:0/2009374742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06880f9920 msgr2=0x7f06880f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f0687fff700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06880f9920 0x7f06880f52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f0687fff700 1 -- 172.21.15.46:0/2009374742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c005040 con 0x7f0688060870 2024-01-21T08:20:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.572+0000 7f0687fff700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688060870 0x7f06880f6e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f06780026d0 tx=0x7f0678009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.573+0000 7f0684ff9700 1 -- 172.21.15.46:0/2009374742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06780079c0 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.573+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06880f5c20 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.573+0000 7f0684ff9700 1 -- 172.21.15.46:0/2009374742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f067800f040 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.573+0000 7f0684ff9700 1 -- 172.21.15.46:0/2009374742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06780148b0 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.573+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068810f130 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.574+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0688045ad0 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.574+0000 7f0684ff9700 1 -- 172.21.15.46:0/2009374742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0678007b20 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.574+0000 7f0684ff9700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f067005c2a0 0x7f067005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.574+0000 7f0684ff9700 1 -- 172.21.15.46:0/2009374742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0678075b00 con 0x7f0688060870 2024-01-21T08:20:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.574+0000 7f06877fe700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f067005c2a0 0x7f067005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.575+0000 7f06877fe700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f067005c2a0 0x7f067005e760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f067c004e30 tx=0x7f067c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.578+0000 7f0684ff9700 1 -- 172.21.15.46:0/2009374742 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0678048b00 con 0x7f0688060870 2024-01-21T08:20:47.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:46.122767+0000 mgr.smithi046.lmxnwc (mgr.14164) 521 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.797+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0688027a00 con 0x7f0688060870 2024-01-21T08:20:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.797+0000 7f0684ff9700 1 -- 172.21.15.46:0/2009374742 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f067804c120 con 0x7f0688060870 2024-01-21T08:20:47.799 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f067005c2a0 msgr2=0x7f067005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f067005c2a0 0x7f067005e760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f067c004e30 tx=0x7f067c009b30).stop 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688060870 msgr2=0x7f06880f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688060870 0x7f06880f6e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f06780026d0 tx=0x7f0678009de0).stop 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 shutdown_connections 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f067005c2a0 0x7f067005e760 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688060870 0x7f06880f6e20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 --2- 172.21.15.46:0/2009374742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06880f9920 0x7f06880f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 >> 172.21.15.46:0/2009374742 conn(0x7f06880f0e20 msgr2=0x7f06880f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 shutdown_connections 2024-01-21T08:20:47.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:47.799+0000 7f068e4f1700 1 -- 172.21.15.46:0/2009374742 wait complete. 2024-01-21T08:20:48.229 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:48.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:48 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:47.798403+0000 mon.smithi046 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.46:0/2009374742' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:48.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:48 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:47.798403+0000 mon.smithi046 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.46:0/2009374742' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:49.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:48.123224+0000 mgr.smithi046.lmxnwc (mgr.14164) 522 : cluster [DBG] pgmap v490: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:49.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:48.123224+0000 mgr.smithi046.lmxnwc (mgr.14164) 522 : cluster [DBG] pgmap v490: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:50.123683+0000 mgr.smithi046.lmxnwc (mgr.14164) 523 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:50.123683+0000 mgr.smithi046.lmxnwc (mgr.14164) 523 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:51.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:51.479 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:52.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 -- 172.21.15.46:0/325711355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 msgr2=0x7f36540f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 --2- 172.21.15.46:0/325711355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36540f6dc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f363c004660 tx=0x7f363c009b30).stop 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 -- 172.21.15.46:0/325711355 shutdown_connections 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 --2- 172.21.15.46:0/325711355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36540f6dc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 --2- 172.21.15.46:0/325711355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36540f56c0 0x7f36540f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 -- 172.21.15.46:0/325711355 >> 172.21.15.46:0/325711355 conn(0x7f36540f0e20 msgr2=0x7f36540f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 -- 172.21.15.46:0/325711355 shutdown_connections 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.440+0000 7f3658e39700 1 -- 172.21.15.46:0/325711355 wait complete. 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3658e39700 1 Processor -- start 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3658e39700 1 -- start start 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3658e39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36540f56c0 0x7f36541070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3658e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36541075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3658e39700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654107b60 con 0x7f36540f6960 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3658e39700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654107ca0 con 0x7f36540f56c0 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3651d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36541075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3651d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36541075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41186/0 (socket says 172.21.15.46:41186) 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f3651d9b700 1 -- 172.21.15.46:0/55184543 learned_addr learned my addr 172.21.15.46:0/55184543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.441+0000 7f365259c700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36540f56c0 0x7f36541070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f3651d9b700 1 -- 172.21.15.46:0/55184543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36540f56c0 msgr2=0x7f36541070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f3651d9b700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36540f56c0 0x7f36541070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f3651d9b700 1 -- 172.21.15.46:0/55184543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c005040 con 0x7f36540f6960 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f3651d9b700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36541075b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f363c009f50 tx=0x7f363c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f364affd700 1 -- 172.21.15.46:0/55184543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363c018070 con 0x7f36540f6960 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f364affd700 1 -- 172.21.15.46:0/55184543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f363c013410 con 0x7f36540f6960 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f364affd700 1 -- 172.21.15.46:0/55184543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363c01c3c0 con 0x7f36540f6960 2024-01-21T08:20:52.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36541085e0 con 0x7f36540f6960 2024-01-21T08:20:52.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.442+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3654108b40 con 0x7f36540f6960 2024-01-21T08:20:52.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.443+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3654045ad0 con 0x7f36540f6960 2024-01-21T08:20:52.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.443+0000 7f364affd700 1 -- 172.21.15.46:0/55184543 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f363c01d420 con 0x7f36540f6960 2024-01-21T08:20:52.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.444+0000 7f364affd700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f364005c2a0 0x7f364005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.444+0000 7f364affd700 1 -- 172.21.15.46:0/55184543 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f363c077730 con 0x7f36540f6960 2024-01-21T08:20:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.444+0000 7f365259c700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f364005c2a0 0x7f364005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.444+0000 7f365259c700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f364005c2a0 0x7f364005e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f36540f67c0 tx=0x7f364400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:52.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.448+0000 7f364affd700 1 -- 172.21.15.46:0/55184543 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f363c04a7b0 con 0x7f36540f6960 2024-01-21T08:20:52.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.666+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f36540fdf00 con 0x7f36540f6960 2024-01-21T08:20:52.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.667+0000 7f364affd700 1 -- 172.21.15.46:0/55184543 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f363c04ddd0 con 0x7f36540f6960 2024-01-21T08:20:52.668 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f364005c2a0 msgr2=0x7f364005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f364005c2a0 0x7f364005e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f36540f67c0 tx=0x7f364400b040).stop 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 msgr2=0x7f36541075b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36541075b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f363c009f50 tx=0x7f363c009b30).stop 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 shutdown_connections 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f364005c2a0 0x7f364005e760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36540f56c0 0x7f36541070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 --2- 172.21.15.46:0/55184543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36540f6960 0x7f36541075b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.668+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 >> 172.21.15.46:0/55184543 conn(0x7f36540f0e20 msgr2=0x7f365405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.669+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 shutdown_connections 2024-01-21T08:20:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:52.669+0000 7f3658e39700 1 -- 172.21.15.46:0/55184543 wait complete. 2024-01-21T08:20:53.078 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:53.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:52.124015+0000 mgr.smithi046.lmxnwc (mgr.14164) 524 : cluster [DBG] pgmap v492: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:53.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:52.667765+0000 mon.smithi046 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.46:0/55184543' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:53.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:52.124015+0000 mgr.smithi046.lmxnwc (mgr.14164) 524 : cluster [DBG] pgmap v492: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:53.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:52.667765+0000 mon.smithi046 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.46:0/55184543' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:55.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:54.124423+0000 mgr.smithi046.lmxnwc (mgr.14164) 525 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:55.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:54.124423+0000 mgr.smithi046.lmxnwc (mgr.14164) 525 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:56.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:20:56.324 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- 172.21.15.46:0/736699573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f67d0 msgr2=0x7fe7480f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/736699573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f67d0 0x7fe7480f6bf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fe744004660 tx=0x7fe744009b30).stop 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- 172.21.15.46:0/736699573 shutdown_connections 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/736699573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7480f7a70 0x7fe7480f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/736699573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f67d0 0x7fe7480f6bf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- 172.21.15.46:0/736699573 >> 172.21.15.46:0/736699573 conn(0x7fe7480f1f50 msgr2=0x7fe7480f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- 172.21.15.46:0/736699573 shutdown_connections 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- 172.21.15.46:0/736699573 wait complete. 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 Processor -- start 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- start start 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7480f67d0 0x7fe74806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f7a70 0x7fe748069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74806a380 con 0x7fe7480f7a70 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74f4ea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74806a4c0 con 0x7fe7480f67d0 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74ca85700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f7a70 0x7fe748069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.311+0000 7fe74ca85700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f7a70 0x7fe748069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41204/0 (socket says 172.21.15.46:41204) 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74ca85700 1 -- 172.21.15.46:0/2995179665 learned_addr learned my addr 172.21.15.46:0/2995179665 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74d286700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7480f67d0 0x7fe74806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74ca85700 1 -- 172.21.15.46:0/2995179665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7480f67d0 msgr2=0x7fe74806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74ca85700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7480f67d0 0x7fe74806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74ca85700 1 -- 172.21.15.46:0/2995179665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe744005040 con 0x7fe7480f7a70 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74ca85700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f7a70 0x7fe748069e20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fe73c0026d0 tx=0x7fe73c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe739ffb700 1 -- 172.21.15.46:0/2995179665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe73c0079c0 con 0x7fe7480f7a70 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe739ffb700 1 -- 172.21.15.46:0/2995179665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe73c00f040 con 0x7fe7480f7a70 2024-01-21T08:20:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74806a7a0 con 0x7fe7480f7a70 2024-01-21T08:20:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe739ffb700 1 -- 172.21.15.46:0/2995179665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe73c0148b0 con 0x7fe7480f7a70 2024-01-21T08:20:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.312+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74810f0e0 con 0x7fe7480f7a70 2024-01-21T08:20:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.313+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe748045ad0 con 0x7fe7480f7a70 2024-01-21T08:20:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.314+0000 7fe739ffb700 1 -- 172.21.15.46:0/2995179665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe73c007b20 con 0x7fe7480f7a70 2024-01-21T08:20:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.314+0000 7fe739ffb700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe73405c2a0 0x7fe73405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:20:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.314+0000 7fe739ffb700 1 -- 172.21.15.46:0/2995179665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe73c075b40 con 0x7fe7480f7a70 2024-01-21T08:20:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.314+0000 7fe74d286700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe73405c2a0 0x7fe73405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:20:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.314+0000 7fe74d286700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe73405c2a0 0x7fe73405e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe74806b730 tx=0x7fe744009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:20:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.318+0000 7fe739ffb700 1 -- 172.21.15.46:0/2995179665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe73c048b40 con 0x7fe7480f7a70 2024-01-21T08:20:57.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.537+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe748027a00 con 0x7fe7480f7a70 2024-01-21T08:20:57.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.538+0000 7fe739ffb700 1 -- 172.21.15.46:0/2995179665 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fe73c04c160 con 0x7fe7480f7a70 2024-01-21T08:20:57.539 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe73405c2a0 msgr2=0x7fe73405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe73405c2a0 0x7fe73405e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe74806b730 tx=0x7fe744009b30).stop 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f7a70 msgr2=0x7fe748069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f7a70 0x7fe748069e20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fe73c0026d0 tx=0x7fe73c009de0).stop 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 shutdown_connections 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe73405c2a0 0x7fe73405e760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7480f67d0 0x7fe74806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 --2- 172.21.15.46:0/2995179665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480f7a70 0x7fe748069e20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.539+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 >> 172.21.15.46:0/2995179665 conn(0x7fe7480f1f50 msgr2=0x7fe7480fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:20:57.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.540+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 shutdown_connections 2024-01-21T08:20:57.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:20:57.540+0000 7fe74f4ea700 1 -- 172.21.15.46:0/2995179665 wait complete. 2024-01-21T08:20:57.706 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:56.124754+0000 mgr.smithi046.lmxnwc (mgr.14164) 526 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:57.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:56.124754+0000 mgr.smithi046.lmxnwc (mgr.14164) 526 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:57.918 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:20:58.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:20:57.538776+0000 mon.smithi046 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.46:0/2995179665' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:58.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:20:57.538776+0000 mon.smithi046 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.46:0/2995179665' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:20:59.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:20:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:20:58.125056+0000 mgr.smithi046.lmxnwc (mgr.14164) 527 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:20:59.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:20:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:20:58.125056+0000 mgr.smithi046.lmxnwc (mgr.14164) 527 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:00.125543+0000 mgr.smithi046.lmxnwc (mgr.14164) 528 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:00.125543+0000 mgr.smithi046.lmxnwc (mgr.14164) 528 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:00.920 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:01.163 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 -- 172.21.15.46:0/609203419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900f9930 msgr2=0x7efc900fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 --2- 172.21.15.46:0/609203419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900f9930 0x7efc900fbd80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7efc8c004660 tx=0x7efc8c009b30).stop 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 -- 172.21.15.46:0/609203419 shutdown_connections 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 --2- 172.21.15.46:0/609203419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900f9930 0x7efc900fbd80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 --2- 172.21.15.46:0/609203419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc900607d0 0x7efc900f9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 -- 172.21.15.46:0/609203419 >> 172.21.15.46:0/609203419 conn(0x7efc900f0e20 msgr2=0x7efc900f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 -- 172.21.15.46:0/609203419 shutdown_connections 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.160+0000 7efc97d2b700 1 -- 172.21.15.46:0/609203419 wait complete. 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc97d2b700 1 Processor -- start 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc97d2b700 1 -- start start 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc97d2b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900607d0 0x7efc90107020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc97d2b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc900f9930 0x7efc90107510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc97d2b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc90107ac0 con 0x7efc900607d0 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc97d2b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc90107c00 con 0x7efc900f9930 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc95ac7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900607d0 0x7efc90107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc95ac7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900607d0 0x7efc90107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36334/0 (socket says 172.21.15.46:36334) 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc95ac7700 1 -- 172.21.15.46:0/1970926601 learned_addr learned my addr 172.21.15.46:0/1970926601 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc952c6700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc900f9930 0x7efc90107510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc95ac7700 1 -- 172.21.15.46:0/1970926601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc900f9930 msgr2=0x7efc90107510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc95ac7700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc900f9930 0x7efc90107510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc95ac7700 1 -- 172.21.15.46:0/1970926601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c005040 con 0x7efc900607d0 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.161+0000 7efc95ac7700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900607d0 0x7efc90107020 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7efc80002700 tx=0x7efc80009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.162+0000 7efc867fc700 1 -- 172.21.15.46:0/1970926601 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc80009320 con 0x7efc900607d0 2024-01-21T08:21:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.162+0000 7efc867fc700 1 -- 172.21.15.46:0/1970926601 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc8000f040 con 0x7efc900607d0 2024-01-21T08:21:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.162+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc901085a0 con 0x7efc900607d0 2024-01-21T08:21:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.162+0000 7efc867fc700 1 -- 172.21.15.46:0/1970926601 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc80014870 con 0x7efc900607d0 2024-01-21T08:21:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.162+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc90108b60 con 0x7efc900607d0 2024-01-21T08:21:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.163+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc90045ad0 con 0x7efc900607d0 2024-01-21T08:21:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.163+0000 7efc867fc700 1 -- 172.21.15.46:0/1970926601 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efc80009480 con 0x7efc900607d0 2024-01-21T08:21:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.163+0000 7efc867fc700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc7c05c2a0 0x7efc7c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.164+0000 7efc867fc700 1 -- 172.21.15.46:0/1970926601 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7efc80075e10 con 0x7efc900607d0 2024-01-21T08:21:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.164+0000 7efc952c6700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc7c05c2a0 0x7efc7c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.164+0000 7efc952c6700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc7c05c2a0 0x7efc7c05e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7efc8c009f50 tx=0x7efc8c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.167+0000 7efc867fc700 1 -- 172.21.15.46:0/1970926601 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc80048e10 con 0x7efc900607d0 2024-01-21T08:21:02.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.384+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc900ff0a0 con 0x7efc900607d0 2024-01-21T08:21:02.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.384+0000 7efc867fc700 1 -- 172.21.15.46:0/1970926601 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7efc8004c430 con 0x7efc900607d0 2024-01-21T08:21:02.386 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc7c05c2a0 msgr2=0x7efc7c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc7c05c2a0 0x7efc7c05e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7efc8c009f50 tx=0x7efc8c009b30).stop 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900607d0 msgr2=0x7efc90107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900607d0 0x7efc90107020 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7efc80002700 tx=0x7efc80009e00).stop 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 shutdown_connections 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7efc7c05c2a0 0x7efc7c05e760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc900607d0 0x7efc90107020 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 --2- 172.21.15.46:0/1970926601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc900f9930 0x7efc90107510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 >> 172.21.15.46:0/1970926601 conn(0x7efc900f0e20 msgr2=0x7efc900f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:02.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 shutdown_connections 2024-01-21T08:21:02.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:02.386+0000 7efc97d2b700 1 -- 172.21.15.46:0/1970926601 wait complete. 2024-01-21T08:21:02.769 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:03.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:02.125878+0000 mgr.smithi046.lmxnwc (mgr.14164) 529 : cluster [DBG] pgmap v497: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:03.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:02.385335+0000 mon.smithi046 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.46:0/1970926601' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:03.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:02.125878+0000 mgr.smithi046.lmxnwc (mgr.14164) 529 : cluster [DBG] pgmap v497: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:03.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:02.385335+0000 mon.smithi046 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.46:0/1970926601' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:05.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:04.126170+0000 mgr.smithi046.lmxnwc (mgr.14164) 530 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:05.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:04.126170+0000 mgr.smithi046.lmxnwc (mgr.14164) 530 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:05.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:06.011 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 -- 172.21.15.46:0/4272996799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 msgr2=0x7fe80c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 --2- 172.21.15.46:0/4272996799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c0f6dc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe808004660 tx=0x7fe808009b30).stop 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 -- 172.21.15.46:0/4272996799 shutdown_connections 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 --2- 172.21.15.46:0/4272996799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c0f6dc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 --2- 172.21.15.46:0/4272996799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe80c0f56c0 0x7fe80c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 -- 172.21.15.46:0/4272996799 >> 172.21.15.46:0/4272996799 conn(0x7fe80c0f0e20 msgr2=0x7fe80c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 -- 172.21.15.46:0/4272996799 shutdown_connections 2024-01-21T08:21:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 -- 172.21.15.46:0/4272996799 wait complete. 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 Processor -- start 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.951+0000 7fe8141cd700 1 -- start start 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8141cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe80c0f56c0 0x7fe80c1070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8141cd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c1075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8141cd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c107b70 con 0x7fe80c0f6960 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8141cd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c107cb0 con 0x7fe80c0f56c0 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811768700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c1075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811768700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c1075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36342/0 (socket says 172.21.15.46:36342) 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811768700 1 -- 172.21.15.46:0/2151559694 learned_addr learned my addr 172.21.15.46:0/2151559694 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811f69700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe80c0f56c0 0x7fe80c1070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811768700 1 -- 172.21.15.46:0/2151559694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe80c0f56c0 msgr2=0x7fe80c1070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811768700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe80c0f56c0 0x7fe80c1070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811768700 1 -- 172.21.15.46:0/2151559694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe808005040 con 0x7fe80c0f6960 2024-01-21T08:21:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe811768700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c1075c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fe808009f50 tx=0x7fe808009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8027fc700 1 -- 172.21.15.46:0/2151559694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe808018070 con 0x7fe80c0f6960 2024-01-21T08:21:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8027fc700 1 -- 172.21.15.46:0/2151559694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe808013410 con 0x7fe80c0f6960 2024-01-21T08:21:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c1085f0 con 0x7fe80c0f6960 2024-01-21T08:21:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8027fc700 1 -- 172.21.15.46:0/2151559694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe80801c3c0 con 0x7fe80c0f6960 2024-01-21T08:21:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.952+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c108b50 con 0x7fe80c0f6960 2024-01-21T08:21:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.953+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80c045ad0 con 0x7fe80c0f6960 2024-01-21T08:21:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.954+0000 7fe8027fc700 1 -- 172.21.15.46:0/2151559694 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe80801d420 con 0x7fe80c0f6960 2024-01-21T08:21:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.954+0000 7fe8027fc700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe7f805c2a0 0x7fe7f805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.954+0000 7fe8027fc700 1 -- 172.21.15.46:0/2151559694 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe808077730 con 0x7fe80c0f6960 2024-01-21T08:21:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.954+0000 7fe811f69700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe7f805c2a0 0x7fe7f805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.954+0000 7fe811f69700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe7f805c2a0 0x7fe7f805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fe80c0f67c0 tx=0x7fe7fc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:06.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:06.958+0000 7fe8027fc700 1 -- 172.21.15.46:0/2151559694 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe80804a7b0 con 0x7fe80c0f6960 2024-01-21T08:21:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.178+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe80c0fdf00 con 0x7fe80c0f6960 2024-01-21T08:21:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.179+0000 7fe8027fc700 1 -- 172.21.15.46:0/2151559694 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fe80804ddd0 con 0x7fe80c0f6960 2024-01-21T08:21:07.180 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe7f805c2a0 msgr2=0x7fe7f805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe7f805c2a0 0x7fe7f805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fe80c0f67c0 tx=0x7fe7fc00b040).stop 2024-01-21T08:21:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 msgr2=0x7fe80c1075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c1075c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fe808009f50 tx=0x7fe808009b30).stop 2024-01-21T08:21:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 shutdown_connections 2024-01-21T08:21:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fe7f805c2a0 0x7fe7f805e760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe80c0f56c0 0x7fe80c1070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 --2- 172.21.15.46:0/2151559694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c0f6960 0x7fe80c1075c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.180+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 >> 172.21.15.46:0/2151559694 conn(0x7fe80c0f0e20 msgr2=0x7fe80c0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.181+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 shutdown_connections 2024-01-21T08:21:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:07.181+0000 7fe8141cd700 1 -- 172.21.15.46:0/2151559694 wait complete. 2024-01-21T08:21:07.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:06.126673+0000 mgr.smithi046.lmxnwc (mgr.14164) 531 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:07.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:07.179799+0000 mon.smithi046 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.46:0/2151559694' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:07.576 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:07.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:06.126673+0000 mgr.smithi046.lmxnwc (mgr.14164) 531 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:07.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:07.179799+0000 mon.smithi046 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.46:0/2151559694' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:08.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:08 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:08.127245+0000 mgr.smithi046.lmxnwc (mgr.14164) 532 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:08.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:08 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:08.127245+0000 mgr.smithi046.lmxnwc (mgr.14164) 532 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:10.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:11.672 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:10.127795+0000 mgr.smithi046.lmxnwc (mgr.14164) 533 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:10.127795+0000 mgr.smithi046.lmxnwc (mgr.14164) 533 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.870+0000 7fbad2aca700 1 -- 172.21.15.46:0/3105747644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 msgr2=0x7fbacc0f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.870+0000 7fbad2aca700 1 --2- 172.21.15.46:0/3105747644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc0f5b00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fbac8004660 tx=0x7fbac8009b30).stop 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 -- 172.21.15.46:0/3105747644 shutdown_connections 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 --2- 172.21.15.46:0/3105747644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbacc0f6980 0x7fbacc0f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 --2- 172.21.15.46:0/3105747644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc0f5b00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 -- 172.21.15.46:0/3105747644 >> 172.21.15.46:0/3105747644 conn(0x7fbacc0f0e20 msgr2=0x7fbacc0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 -- 172.21.15.46:0/3105747644 shutdown_connections 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 -- 172.21.15.46:0/3105747644 wait complete. 2024-01-21T08:21:11.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 Processor -- start 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 -- start start 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbacc0f6980 0x7fbacc107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc107b20 con 0x7fbacc0f56e0 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.871+0000 7fbad2aca700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc107c60 con 0x7fbacc0f6980 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad0866700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad0866700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42398/0 (socket says 172.21.15.46:42398) 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad0866700 1 -- 172.21.15.46:0/287379462 learned_addr learned my addr 172.21.15.46:0/287379462 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbac3fff700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbacc0f6980 0x7fbacc107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad0866700 1 -- 172.21.15.46:0/287379462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbacc0f6980 msgr2=0x7fbacc107570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad0866700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbacc0f6980 0x7fbacc107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad0866700 1 -- 172.21.15.46:0/287379462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8005040 con 0x7fbacc0f56e0 2024-01-21T08:21:11.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad0866700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc107080 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbac8000c00 tx=0x7fbac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:11.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbac17fa700 1 -- 172.21.15.46:0/287379462 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac8019070 con 0x7fbacc0f56e0 2024-01-21T08:21:11.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbac17fa700 1 -- 172.21.15.46:0/287379462 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbac8007e70 con 0x7fbacc0f56e0 2024-01-21T08:21:11.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbac17fa700 1 -- 172.21.15.46:0/287379462 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac8014db0 con 0x7fbacc0f56e0 2024-01-21T08:21:11.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc1085a0 con 0x7fbacc0f56e0 2024-01-21T08:21:11.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.872+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc108b00 con 0x7fbacc0f56e0 2024-01-21T08:21:11.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.873+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc045ad0 con 0x7fbacc0f56e0 2024-01-21T08:21:11.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.873+0000 7fbac17fa700 1 -- 172.21.15.46:0/287379462 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbac8025070 con 0x7fbacc0f56e0 2024-01-21T08:21:11.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.874+0000 7fbac17fa700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbab405c240 0x7fbab405e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:11.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.874+0000 7fbac17fa700 1 -- 172.21.15.46:0/287379462 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbac8076ac0 con 0x7fbacc0f56e0 2024-01-21T08:21:11.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.874+0000 7fbac3fff700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbab405c240 0x7fbab405e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:11.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.874+0000 7fbac3fff700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbab405c240 0x7fbab405e700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbabc009da0 tx=0x7fbabc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:11.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:11.878+0000 7fbac17fa700 1 -- 172.21.15.46:0/287379462 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbac8049ac0 con 0x7fbacc0f56e0 2024-01-21T08:21:12.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.096+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbacc109190 con 0x7fbacc0f56e0 2024-01-21T08:21:12.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.097+0000 7fbac17fa700 1 -- 172.21.15.46:0/287379462 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fbac804d0e0 con 0x7fbacc0f56e0 2024-01-21T08:21:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbab405c240 msgr2=0x7fbab405e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbab405c240 0x7fbab405e700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbabc009da0 tx=0x7fbabc00b040).stop 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 msgr2=0x7fbacc107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc107080 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbac8000c00 tx=0x7fbac800b040).stop 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 shutdown_connections 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbab405c240 0x7fbab405e700 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0f56e0 0x7fbacc107080 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 --2- 172.21.15.46:0/287379462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbacc0f6980 0x7fbacc107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 >> 172.21.15.46:0/287379462 conn(0x7fbacc0f0e20 msgr2=0x7fbacc0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:12.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 shutdown_connections 2024-01-21T08:21:12.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:12.098+0000 7fbad2aca700 1 -- 172.21.15.46:0/287379462 wait complete. 2024-01-21T08:21:12.469 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:12.098633+0000 mon.smithi046 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.46:0/287379462' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:12.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:12 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:12.098633+0000 mon.smithi046 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.46:0/287379462' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:13.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:12.128112+0000 mgr.smithi046.lmxnwc (mgr.14164) 534 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:13.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:12.128112+0000 mgr.smithi046.lmxnwc (mgr.14164) 534 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:15.470 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:15.716 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:15.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:14.128507+0000 mgr.smithi046.lmxnwc (mgr.14164) 535 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:15.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:14.128507+0000 mgr.smithi046.lmxnwc (mgr.14164) 535 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:16.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.623+0000 7f40ea796700 1 -- 172.21.15.46:0/2266750704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f56c0 msgr2=0x7f40e40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:16.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.623+0000 7f40ea796700 1 --2- 172.21.15.46:0/2266750704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f56c0 0x7f40e40f5ae0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f40d4004660 tx=0x7f40d4009b30).stop 2024-01-21T08:21:16.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.623+0000 7f40ea796700 1 -- 172.21.15.46:0/2266750704 shutdown_connections 2024-01-21T08:21:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.623+0000 7f40ea796700 1 --2- 172.21.15.46:0/2266750704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40f6960 0x7f40e40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:21:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.623+0000 7f40ea796700 1 --2- 172.21.15.46:0/2266750704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f56c0 0x7f40e40f5ae0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.623+0000 7f40ea796700 1 -- 172.21.15.46:0/2266750704 >> 172.21.15.46:0/2266750704 conn(0x7f40e40f0e20 msgr2=0x7f40e40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.624+0000 7f40ea796700 1 -- 172.21.15.46:0/2266750704 shutdown_connections 2024-01-21T08:21:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.624+0000 7f40ea796700 1 -- 172.21.15.46:0/2266750704 wait complete. 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.624+0000 7f40ea796700 1 Processor -- start 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.624+0000 7f40ea796700 1 -- start start 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40ea796700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40f56c0 0x7f40e4105fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40ea796700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f6960 0x7f40e41064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40ea796700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e4106b00 con 0x7f40e40f6960 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40ea796700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e41073b0 con 0x7f40e40f56c0 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f6960 0x7f40e41064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f6960 0x7f40e41064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42414/0 (socket says 172.21.15.46:42414) 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e37fe700 1 -- 172.21.15.46:0/2167491986 learned_addr learned my addr 172.21.15.46:0/2167491986 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e3fff700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40f56c0 0x7f40e4105fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e37fe700 1 -- 172.21.15.46:0/2167491986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40f56c0 msgr2=0x7f40e4105fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e37fe700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40f56c0 0x7f40e4105fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e37fe700 1 -- 172.21.15.46:0/2167491986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4005040 con 0x7f40e40f6960 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e37fe700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f6960 0x7f40e41064c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f40cc00b410 tx=0x7f40cc006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e0ff9700 1 -- 172.21.15.46:0/2167491986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40cc0125f0 con 0x7f40e40f6960 2024-01-21T08:21:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e0ff9700 1 -- 172.21.15.46:0/2167491986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40cc012c00 con 0x7f40e40f6960 2024-01-21T08:21:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e4107690 con 0x7f40e40f6960 2024-01-21T08:21:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.625+0000 7f40e0ff9700 1 -- 172.21.15.46:0/2167491986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40cc002aa0 con 0x7f40e40f6960 2024-01-21T08:21:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.626+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e4107c50 con 0x7f40e40f6960 2024-01-21T08:21:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.626+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e4059100 con 0x7f40e40f6960 2024-01-21T08:21:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.627+0000 7f40e0ff9700 1 -- 172.21.15.46:0/2167491986 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f40cc00f030 con 0x7f40e40f6960 2024-01-21T08:21:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.627+0000 7f40e0ff9700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f40d005c2a0 0x7f40d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.627+0000 7f40e0ff9700 1 -- 172.21.15.46:0/2167491986 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f40cc075f10 con 0x7f40e40f6960 2024-01-21T08:21:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.627+0000 7f40e3fff700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f40d005c2a0 0x7f40d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.628+0000 7f40e3fff700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f40d005c2a0 0x7f40d005e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f40d4004e30 tx=0x7f40d4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:16.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.631+0000 7f40e0ff9700 1 -- 172.21.15.46:0/2167491986 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40cc048f30 con 0x7f40e40f6960 2024-01-21T08:21:16.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:16.128857+0000 mgr.smithi046.lmxnwc (mgr.14164) 536 : cluster [DBG] pgmap v504: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.852+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f40e40fded0 con 0x7f40e40f6960 2024-01-21T08:21:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.852+0000 7f40e0ff9700 1 -- 172.21.15.46:0/2167491986 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f40cc04c550 con 0x7f40e40f6960 2024-01-21T08:21:16.855 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f40d005c2a0 msgr2=0x7f40d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f40d005c2a0 0x7f40d005e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f40d4004e30 tx=0x7f40d4009b30).stop 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f6960 msgr2=0x7f40e41064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f6960 0x7f40e41064c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f40cc00b410 tx=0x7f40cc006ef0).stop 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 shutdown_connections 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f40d005c2a0 0x7f40d005e760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40f56c0 0x7f40e4105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 --2- 172.21.15.46:0/2167491986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e40f6960 0x7f40e41064c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 >> 172.21.15.46:0/2167491986 conn(0x7f40e40f0e20 msgr2=0x7f40e40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 shutdown_connections 2024-01-21T08:21:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:16.854+0000 7f40ea796700 1 -- 172.21.15.46:0/2167491986 wait complete. 2024-01-21T08:21:16.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:16.128857+0000 mgr.smithi046.lmxnwc (mgr.14164) 536 : cluster [DBG] pgmap v504: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:17.225 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:17.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:16.854308+0000 mon.smithi046 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.46:0/2167491986' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:17.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:16.854308+0000 mon.smithi046 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.46:0/2167491986' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:18.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:18 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:18.129364+0000 mgr.smithi046.lmxnwc (mgr.14164) 537 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:18.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:18 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:18.129364+0000 mgr.smithi046.lmxnwc (mgr.14164) 537 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:20.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:20.470 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.407+0000 7f88f5c3b700 1 -- 172.21.15.46:0/2694395628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f67b0 msgr2=0x7f88f00f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.407+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/2694395628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f67b0 0x7f88f00f6c10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f88dc004660 tx=0x7f88dc009b30).stop 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.407+0000 7f88f5c3b700 1 -- 172.21.15.46:0/2694395628 shutdown_connections 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.407+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/2694395628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f67b0 0x7f88f00f6c10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.407+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/2694395628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f00f5510 0x7f88f00f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.407+0000 7f88f5c3b700 1 -- 172.21.15.46:0/2694395628 >> 172.21.15.46:0/2694395628 conn(0x7f88f00f0c70 msgr2=0x7f88f00f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 -- 172.21.15.46:0/2694395628 shutdown_connections 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 -- 172.21.15.46:0/2694395628 wait complete. 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 Processor -- start 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 -- start start 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f5510 0x7f88f00f9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:21.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f00f67b0 0x7f88f00f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00f8050 con 0x7f88f00f5510 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f5c3b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00f8190 con 0x7f88f00f67b0 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.408+0000 7f88f4c39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f5510 0x7f88f00f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88f4c39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f5510 0x7f88f00f9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50180/0 (socket says 172.21.15.46:50180) 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88f4c39700 1 -- 172.21.15.46:0/1926968023 learned_addr learned my addr 172.21.15.46:0/1926968023 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88effff700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f00f67b0 0x7f88f00f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88f4c39700 1 -- 172.21.15.46:0/1926968023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f00f67b0 msgr2=0x7f88f00f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88f4c39700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f00f67b0 0x7f88f00f7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88f4c39700 1 -- 172.21.15.46:0/1926968023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc005040 con 0x7f88f00f5510 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88f4c39700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f5510 0x7f88f00f9670 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f88e4002700 tx=0x7f88e4009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88ed7fa700 1 -- 172.21.15.46:0/1926968023 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e4009320 con 0x7f88f00f5510 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f00f8470 con 0x7f88f00f5510 2024-01-21T08:21:21.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88ed7fa700 1 -- 172.21.15.46:0/1926968023 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88e400f040 con 0x7f88f00f5510 2024-01-21T08:21:21.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.409+0000 7f88ed7fa700 1 -- 172.21.15.46:0/1926968023 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e4014900 con 0x7f88f00f5510 2024-01-21T08:21:21.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.410+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f010ef00 con 0x7f88f00f5510 2024-01-21T08:21:21.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.411+0000 7f88ed7fa700 1 -- 172.21.15.46:0/1926968023 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f88e4009480 con 0x7f88f00f5510 2024-01-21T08:21:21.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.411+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f0045ad0 con 0x7f88f00f5510 2024-01-21T08:21:21.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.411+0000 7f88ed7fa700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f88e005c2b0 0x7f88e005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:21.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.411+0000 7f88ed7fa700 1 -- 172.21.15.46:0/1926968023 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f88e4075df0 con 0x7f88f00f5510 2024-01-21T08:21:21.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.411+0000 7f88effff700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f88e005c2b0 0x7f88e005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:21.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.412+0000 7f88effff700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f88e005c2b0 0x7f88e005e770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f88dc004e30 tx=0x7f88dc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.415+0000 7f88ed7fa700 1 -- 172.21.15.46:0/1926968023 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88e4048d30 con 0x7f88f00f5510 2024-01-21T08:21:21.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:20.129868+0000 mgr.smithi046.lmxnwc (mgr.14164) 538 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:21.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:20.763427+0000 mon.smithi046 (mon.0) 549 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:21:21.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:20.764183+0000 mon.smithi046 (mon.0) 550 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:21:21.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.640+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f88f00fdca0 con 0x7f88f00f5510 2024-01-21T08:21:21.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.640+0000 7f88ed7fa700 1 -- 172.21.15.46:0/1926968023 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f88e404c350 con 0x7f88f00f5510 2024-01-21T08:21:21.642 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:21.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f88e005c2b0 msgr2=0x7f88e005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:21.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f88e005c2b0 0x7f88e005e770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f88dc004e30 tx=0x7f88dc009b30).stop 2024-01-21T08:21:21.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f5510 msgr2=0x7f88f00f9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:21.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f5510 0x7f88f00f9670 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f88e4002700 tx=0x7f88e4009d80).stop 2024-01-21T08:21:21.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 shutdown_connections 2024-01-21T08:21:21.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f88e005c2b0 0x7f88e005e770 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:21.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88f00f5510 0x7f88f00f9670 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:21.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 --2- 172.21.15.46:0/1926968023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f00f67b0 0x7f88f00f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:21.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 >> 172.21.15.46:0/1926968023 conn(0x7f88f00f0c70 msgr2=0x7f88f0060380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:21.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 shutdown_connections 2024-01-21T08:21:21.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:21.642+0000 7f88f5c3b700 1 -- 172.21.15.46:0/1926968023 wait complete. 2024-01-21T08:21:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:20.129868+0000 mgr.smithi046.lmxnwc (mgr.14164) 538 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:20.763427+0000 mon.smithi046 (mon.0) 549 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:21:21.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:20.764183+0000 mon.smithi046 (mon.0) 550 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:21:22.033 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:21.641860+0000 mon.smithi046 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.46:0/1926968023' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:22.130134+0000 mgr.smithi046.lmxnwc (mgr.14164) 539 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:21.641860+0000 mon.smithi046 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.46:0/1926968023' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:22.130134+0000 mgr.smithi046.lmxnwc (mgr.14164) 539 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:25.034 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:25.280 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:25.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:24.130587+0000 mgr.smithi046.lmxnwc (mgr.14164) 540 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:25.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:24.130587+0000 mgr.smithi046.lmxnwc (mgr.14164) 540 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 -- 172.21.15.46:0/2166424780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 msgr2=0x7f42fc0f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 --2- 172.21.15.46:0/2166424780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc0f7d10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f42ec004660 tx=0x7f42ec009b30).stop 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 -- 172.21.15.46:0/2166424780 shutdown_connections 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 --2- 172.21.15.46:0/2166424780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc0f7d10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 --2- 172.21.15.46:0/2166424780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc0f6610 0x7f42fc0f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 -- 172.21.15.46:0/2166424780 >> 172.21.15.46:0/2166424780 conn(0x7f42fc0f1d70 msgr2=0x7f42fc0f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 -- 172.21.15.46:0/2166424780 shutdown_connections 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 -- 172.21.15.46:0/2166424780 wait complete. 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 Processor -- start 2024-01-21T08:21:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.290+0000 7f4300c8e700 1 -- start start 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f4300c8e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc0f6610 0x7f42fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f4300c8e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f4300c8e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc06a380 con 0x7f42fc0f78b0 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f4300c8e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc06a4c0 con 0x7f42fc0f6610 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42faffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42faffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50198/0 (socket says 172.21.15.46:50198) 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42faffd700 1 -- 172.21.15.46:0/1318483499 learned_addr learned my addr 172.21.15.46:0/1318483499 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:26.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42fb7fe700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc0f6610 0x7f42fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42faffd700 1 -- 172.21.15.46:0/1318483499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc0f6610 msgr2=0x7f42fc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42faffd700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc0f6610 0x7f42fc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42faffd700 1 -- 172.21.15.46:0/1318483499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec005040 con 0x7f42fc0f78b0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42faffd700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc069e20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f42ec009990 tx=0x7f42ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42f3fff700 1 -- 172.21.15.46:0/1318483499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ec018070 con 0x7f42fc0f78b0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc06a740 con 0x7f42fc0f78b0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42f3fff700 1 -- 172.21.15.46:0/1318483499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42ec013410 con 0x7f42fc0f78b0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f42f3fff700 1 -- 172.21.15.46:0/1318483499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ec01c3c0 con 0x7f42fc0f78b0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.291+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc06aca0 con 0x7f42fc0f78b0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.292+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc045ad0 con 0x7f42fc0f78b0 2024-01-21T08:21:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.293+0000 7f42f3fff700 1 -- 172.21.15.46:0/1318483499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f42ec01d420 con 0x7f42fc0f78b0 2024-01-21T08:21:26.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.293+0000 7f42f3fff700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f42e805c2a0 0x7f42e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:26.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.293+0000 7f42f3fff700 1 -- 172.21.15.46:0/1318483499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f42ec077900 con 0x7f42fc0f78b0 2024-01-21T08:21:26.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.293+0000 7f42fb7fe700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f42e805c2a0 0x7f42e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:26.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.293+0000 7f42fb7fe700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f42e805c2a0 0x7f42e805e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f42fc06b730 tx=0x7f42e4009c00).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:26.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.297+0000 7f42f3fff700 1 -- 172.21.15.46:0/1318483499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42ec04a8a0 con 0x7f42fc0f78b0 2024-01-21T08:21:26.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.516+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f42fc027a00 con 0x7f42fc0f78b0 2024-01-21T08:21:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.517+0000 7f42f3fff700 1 -- 172.21.15.46:0/1318483499 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f42ec04dec0 con 0x7f42fc0f78b0 2024-01-21T08:21:26.520 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f42e805c2a0 msgr2=0x7f42e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f42e805c2a0 0x7f42e805e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f42fc06b730 tx=0x7f42e4009c00).stop 2024-01-21T08:21:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 msgr2=0x7f42fc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc069e20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f42ec009990 tx=0x7f42ec009b30).stop 2024-01-21T08:21:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 shutdown_connections 2024-01-21T08:21:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f42e805c2a0 0x7f42e805e760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc0f6610 0x7f42fc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 --2- 172.21.15.46:0/1318483499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0f78b0 0x7f42fc069e20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 >> 172.21.15.46:0/1318483499 conn(0x7f42fc0f1d70 msgr2=0x7f42fc0fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 shutdown_connections 2024-01-21T08:21:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:26.519+0000 7f4300c8e700 1 -- 172.21.15.46:0/1318483499 wait complete. 2024-01-21T08:21:26.865 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:26.130881+0000 mgr.smithi046.lmxnwc (mgr.14164) 541 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:26.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:26.130881+0000 mgr.smithi046.lmxnwc (mgr.14164) 541 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:27.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:26.518785+0000 mon.smithi046 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.46:0/1318483499' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:27.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:26.518785+0000 mon.smithi046 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.46:0/1318483499' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:28.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:28 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:28.131273+0000 mgr.smithi046.lmxnwc (mgr.14164) 542 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:28.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:28 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:28.131273+0000 mgr.smithi046.lmxnwc (mgr.14164) 542 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:29.899 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:30.144 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:30.609 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:30.131577+0000 mgr.smithi046.lmxnwc (mgr.14164) 543 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:30.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:30.131577+0000 mgr.smithi046.lmxnwc (mgr.14164) 543 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- 172.21.15.46:0/38358673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80fbd10 msgr2=0x7f08e80fe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 --2- 172.21.15.46:0/38358673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80fbd10 0x7f08e80fe100 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f08dc004690 tx=0x7f08dc009b30).stop 2024-01-21T08:21:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- 172.21.15.46:0/38358673 shutdown_connections 2024-01-21T08:21:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 --2- 172.21.15.46:0/38358673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80fbd10 0x7f08e80fe100 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 --2- 172.21.15.46:0/38358673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e80607f0 0x7f08e80fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- 172.21.15.46:0/38358673 >> 172.21.15.46:0/38358673 conn(0x7f08e80f0e20 msgr2=0x7f08e80f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- 172.21.15.46:0/38358673 shutdown_connections 2024-01-21T08:21:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- 172.21.15.46:0/38358673 wait complete. 2024-01-21T08:21:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 Processor -- start 2024-01-21T08:21:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- start start 2024-01-21T08:21:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80607f0 0x7f08e8109270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e80fbd10 0x7f08e8109760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8109d10 con 0x7f08e80607f0 2024-01-21T08:21:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08ecda3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8109e50 con 0x7f08e80fbd10 2024-01-21T08:21:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.527+0000 7f08e659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80607f0 0x7f08e8109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08e659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80607f0 0x7f08e8109270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50284/0 (socket says 172.21.15.46:50284) 2024-01-21T08:21:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08e659c700 1 -- 172.21.15.46:0/4064465880 learned_addr learned my addr 172.21.15.46:0/4064465880 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08e5d9b700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e80fbd10 0x7f08e8109760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08e659c700 1 -- 172.21.15.46:0/4064465880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e80fbd10 msgr2=0x7f08e8109760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08e659c700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e80fbd10 0x7f08e8109760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08e659c700 1 -- 172.21.15.46:0/4064465880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc005040 con 0x7f08e80607f0 2024-01-21T08:21:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08e659c700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80607f0 0x7f08e8109270 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f08d800c530 tx=0x7f08d8007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08d6ffd700 1 -- 172.21.15.46:0/4064465880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d800f650 con 0x7f08e80607f0 2024-01-21T08:21:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08d6ffd700 1 -- 172.21.15.46:0/4064465880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08d800fc60 con 0x7f08e80607f0 2024-01-21T08:21:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e810a7f0 con 0x7f08e80607f0 2024-01-21T08:21:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08d6ffd700 1 -- 172.21.15.46:0/4064465880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d8012c30 con 0x7f08e80607f0 2024-01-21T08:21:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.528+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e810adb0 con 0x7f08e80607f0 2024-01-21T08:21:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.529+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e8045ad0 con 0x7f08e80607f0 2024-01-21T08:21:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.529+0000 7f08d6ffd700 1 -- 172.21.15.46:0/4064465880 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f08d8010070 con 0x7f08e80607f0 2024-01-21T08:21:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.530+0000 7f08d6ffd700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f08d005c2a0 0x7f08d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.530+0000 7f08d6ffd700 1 -- 172.21.15.46:0/4064465880 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f08d8075730 con 0x7f08e80607f0 2024-01-21T08:21:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.530+0000 7f08e5d9b700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f08d005c2a0 0x7f08d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.530+0000 7f08e5d9b700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f08d005c2a0 0x7f08d005e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f08dc005010 tx=0x7f08dc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:31.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.534+0000 7f08d6ffd700 1 -- 172.21.15.46:0/4064465880 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08d80452b0 con 0x7f08e80607f0 2024-01-21T08:21:31.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.750+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f08e81012a0 con 0x7f08e80607f0 2024-01-21T08:21:31.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.750+0000 7f08d6ffd700 1 -- 172.21.15.46:0/4064465880 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f08d80487b0 con 0x7f08e80607f0 2024-01-21T08:21:31.753 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f08d005c2a0 msgr2=0x7f08d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f08d005c2a0 0x7f08d005e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f08dc005010 tx=0x7f08dc009b30).stop 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80607f0 msgr2=0x7f08e8109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80607f0 0x7f08e8109270 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f08d800c530 tx=0x7f08d8007e20).stop 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 shutdown_connections 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f08d005c2a0 0x7f08d005e760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08e80607f0 0x7f08e8109270 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 --2- 172.21.15.46:0/4064465880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e80fbd10 0x7f08e8109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 >> 172.21.15.46:0/4064465880 conn(0x7f08e80f0e20 msgr2=0x7f08e80f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 shutdown_connections 2024-01-21T08:21:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:31.752+0000 7f08ecda3700 1 -- 172.21.15.46:0/4064465880 wait complete. 2024-01-21T08:21:32.088 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:31 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:31.752397+0000 mon.smithi046 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.46:0/4064465880' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:32.123 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:32.241 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:31 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:31.752397+0000 mon.smithi046 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.46:0/4064465880' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:33.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:32 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:32.132043+0000 mgr.smithi046.lmxnwc (mgr.14164) 544 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:33.241 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:32 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:32.132043+0000 mgr.smithi046.lmxnwc (mgr.14164) 544 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:35.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:35.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:34.132408+0000 mgr.smithi046.lmxnwc (mgr.14164) 545 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:35.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:34.132408+0000 mgr.smithi046.lmxnwc (mgr.14164) 545 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/2558818018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f56f0 msgr2=0x7fd7d40f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/2558818018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f56f0 0x7fd7d40f5b10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fd7c4004660 tx=0x7fd7c4009b30).stop 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/2558818018 shutdown_connections 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/2558818018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7d40f6990 0x7fd7d40f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/2558818018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f56f0 0x7fd7d40f5b10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/2558818018 >> 172.21.15.46:0/2558818018 conn(0x7fd7d40f0e50 msgr2=0x7fd7d40f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/2558818018 shutdown_connections 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.366+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/2558818018 wait complete. 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d8cd9700 1 Processor -- start 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d8cd9700 1 -- start start 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d8cd9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7d40f56f0 0x7fd7d4109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d8cd9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f6990 0x7fd7d410c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d8cd9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d4060920 con 0x7fd7d40f6990 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d8cd9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d4060a60 con 0x7fd7d40f56f0 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f6990 0x7fd7d410c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f6990 0x7fd7d410c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50302/0 (socket says 172.21.15.46:50302) 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d1d9b700 1 -- 172.21.15.46:0/849790210 learned_addr learned my addr 172.21.15.46:0/849790210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.367+0000 7fd7d259c700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7d40f56f0 0x7fd7d4109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7d1d9b700 1 -- 172.21.15.46:0/849790210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7d40f56f0 msgr2=0x7fd7d4109dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7d1d9b700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7d40f56f0 0x7fd7d4109dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7d1d9b700 1 -- 172.21.15.46:0/849790210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4005040 con 0x7fd7d40f6990 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7d1d9b700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f6990 0x7fd7d410c2d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fd7bc00c790 tx=0x7fd7bc007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7caffd700 1 -- 172.21.15.46:0/849790210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7bc002590 con 0x7fd7d40f6990 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7caffd700 1 -- 172.21.15.46:0/849790210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7bc002ba0 con 0x7fd7d40f6990 2024-01-21T08:21:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d410c880 con 0x7fd7d40f6990 2024-01-21T08:21:36.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7caffd700 1 -- 172.21.15.46:0/849790210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7bc013d90 con 0x7fd7d40f6990 2024-01-21T08:21:36.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.368+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d410ce10 con 0x7fd7d40f6990 2024-01-21T08:21:36.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.369+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d4059100 con 0x7fd7d40f6990 2024-01-21T08:21:36.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.369+0000 7fd7caffd700 1 -- 172.21.15.46:0/849790210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd7bc01a060 con 0x7fd7d40f6990 2024-01-21T08:21:36.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.370+0000 7fd7caffd700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd7c005c2a0 0x7fd7c005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:36.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.370+0000 7fd7caffd700 1 -- 172.21.15.46:0/849790210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd7bc075df0 con 0x7fd7d40f6990 2024-01-21T08:21:36.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.370+0000 7fd7d259c700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd7c005c2a0 0x7fd7c005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:36.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.370+0000 7fd7d259c700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd7c005c2a0 0x7fd7c005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fd7d40605a0 tx=0x7fd7c4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:36.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.374+0000 7fd7caffd700 1 -- 172.21.15.46:0/849790210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7bc048d90 con 0x7fd7d40f6990 2024-01-21T08:21:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.594+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd7d40fe010 con 0x7fd7d40f6990 2024-01-21T08:21:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.594+0000 7fd7caffd700 1 -- 172.21.15.46:0/849790210 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fd7d40fe010 con 0x7fd7d40f6990 2024-01-21T08:21:36.597 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:36.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.595+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd7c005c2a0 msgr2=0x7fd7c005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:36.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.595+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd7c005c2a0 0x7fd7c005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fd7d40605a0 tx=0x7fd7c4009b30).stop 2024-01-21T08:21:36.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.595+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f6990 msgr2=0x7fd7d410c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:36.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.595+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f6990 0x7fd7d410c2d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fd7bc00c790 tx=0x7fd7bc007d70).stop 2024-01-21T08:21:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.596+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 shutdown_connections 2024-01-21T08:21:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.596+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd7c005c2a0 0x7fd7c005e760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.596+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7d40f56f0 0x7fd7d4109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.596+0000 7fd7d8cd9700 1 --2- 172.21.15.46:0/849790210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7d40f6990 0x7fd7d410c2d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.596+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 >> 172.21.15.46:0/849790210 conn(0x7fd7d40f0e50 msgr2=0x7fd7d40fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.596+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 shutdown_connections 2024-01-21T08:21:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:36.596+0000 7fd7d8cd9700 1 -- 172.21.15.46:0/849790210 wait complete. 2024-01-21T08:21:36.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:36.132884+0000 mgr.smithi046.lmxnwc (mgr.14164) 546 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:36.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:36 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:36.132884+0000 mgr.smithi046.lmxnwc (mgr.14164) 546 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:36.972 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:37.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:36.596014+0000 mon.smithi046 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.46:0/849790210' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:37.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:36.596014+0000 mon.smithi046 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.46:0/849790210' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:38.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:38 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:38.133303+0000 mgr.smithi046.lmxnwc (mgr.14164) 547 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:38.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:38 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:38.133303+0000 mgr.smithi046.lmxnwc (mgr.14164) 547 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:39.973 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:40.219 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:40.667 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:40.133624+0000 mgr.smithi046.lmxnwc (mgr.14164) 548 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:40.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:40.133624+0000 mgr.smithi046.lmxnwc (mgr.14164) 548 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:41.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 -- 172.21.15.46:0/1720979609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 msgr2=0x7f20880f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 --2- 172.21.15.46:0/1720979609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f20880f6dc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f2074004660 tx=0x7f2074009b30).stop 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 -- 172.21.15.46:0/1720979609 shutdown_connections 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 --2- 172.21.15.46:0/1720979609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f20880f6dc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 --2- 172.21.15.46:0/1720979609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20880f56c0 0x7f20880f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 -- 172.21.15.46:0/1720979609 >> 172.21.15.46:0/1720979609 conn(0x7f20880f0e20 msgr2=0x7f20880f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 -- 172.21.15.46:0/1720979609 shutdown_connections 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 -- 172.21.15.46:0/1720979609 wait complete. 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.163+0000 7f208ea97700 1 Processor -- start 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f208ea97700 1 -- start start 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f208ea97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20880f56c0 0x7f2088105f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f208ea97700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f2088106410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f208ea97700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20881069c0 con 0x7f20880f6960 2024-01-21T08:21:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f208ea97700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088106b00 con 0x7f20880f56c0 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f2087fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f2088106410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f2087fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f2088106410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55362/0 (socket says 172.21.15.46:55362) 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f2087fff700 1 -- 172.21.15.46:0/3613595089 learned_addr learned my addr 172.21.15.46:0/3613595089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f208c833700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20880f56c0 0x7f2088105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f2087fff700 1 -- 172.21.15.46:0/3613595089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20880f56c0 msgr2=0x7f2088105f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f2087fff700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20880f56c0 0x7f2088105f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f2087fff700 1 -- 172.21.15.46:0/3613595089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2074005040 con 0x7f20880f6960 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.164+0000 7f2087fff700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f2088106410 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f2074000c00 tx=0x7f207400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.165+0000 7f20857fa700 1 -- 172.21.15.46:0/3613595089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2074019070 con 0x7f20880f6960 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.165+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2088107440 con 0x7f20880f6960 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.165+0000 7f20857fa700 1 -- 172.21.15.46:0/3613595089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2074007e70 con 0x7f20880f6960 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.165+0000 7f20857fa700 1 -- 172.21.15.46:0/3613595089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2074014c90 con 0x7f20880f6960 2024-01-21T08:21:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.165+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20881079a0 con 0x7f20880f6960 2024-01-21T08:21:41.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.166+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2088045ad0 con 0x7f20880f6960 2024-01-21T08:21:41.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.166+0000 7f20857fa700 1 -- 172.21.15.46:0/3613595089 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2074025070 con 0x7f20880f6960 2024-01-21T08:21:41.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.166+0000 7f20857fa700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f207805c2a0 0x7f207805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:41.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.167+0000 7f20857fa700 1 -- 172.21.15.46:0/3613595089 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2074076be0 con 0x7f20880f6960 2024-01-21T08:21:41.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.167+0000 7f208c833700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f207805c2a0 0x7f207805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:41.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.167+0000 7f208c833700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f207805c2a0 0x7f207805e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f207c009fd0 tx=0x7f207c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:41.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.171+0000 7f20857fa700 1 -- 172.21.15.46:0/3613595089 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2074049be0 con 0x7f20880f6960 2024-01-21T08:21:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.393+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2088027a00 con 0x7f20880f6960 2024-01-21T08:21:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.394+0000 7f20857fa700 1 -- 172.21.15.46:0/3613595089 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f207404d200 con 0x7f20880f6960 2024-01-21T08:21:41.395 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.395+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f207805c2a0 msgr2=0x7f207805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.395+0000 7f208ea97700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f207805c2a0 0x7f207805e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f207c009fd0 tx=0x7f207c00b040).stop 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.395+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 msgr2=0x7f2088106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.395+0000 7f208ea97700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f2088106410 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f2074000c00 tx=0x7f207400b040).stop 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.396+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 shutdown_connections 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.396+0000 7f208ea97700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f207805c2a0 0x7f207805e760 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.396+0000 7f208ea97700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20880f56c0 0x7f2088105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.396+0000 7f208ea97700 1 --2- 172.21.15.46:0/3613595089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20880f6960 0x7f2088106410 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.396+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 >> 172.21.15.46:0/3613595089 conn(0x7f20880f0e20 msgr2=0x7f20880fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:41.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.396+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 shutdown_connections 2024-01-21T08:21:41.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:41.396+0000 7f208ea97700 1 -- 172.21.15.46:0/3613595089 wait complete. 2024-01-21T08:21:41.704 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:41 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:41.394687+0000 mon.smithi046 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.46:0/3613595089' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:41.771 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:41.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:41 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:41.394687+0000 mon.smithi046 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.46:0/3613595089' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:42.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:42 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:42.134105+0000 mgr.smithi046.lmxnwc (mgr.14164) 549 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:42.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:42 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:42.134105+0000 mgr.smithi046.lmxnwc (mgr.14164) 549 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:44.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:45.015 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:45.467 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:44.134670+0000 mgr.smithi046.lmxnwc (mgr.14164) 550 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:45.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:44.134670+0000 mgr.smithi046.lmxnwc (mgr.14164) 550 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:45.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 -- 172.21.15.46:0/2895617895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f56c0 msgr2=0x7f32380f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:45.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2895617895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f56c0 0x7f32380f5ae0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f3228004660 tx=0x7f3228009b30).stop 2024-01-21T08:21:45.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 -- 172.21.15.46:0/2895617895 shutdown_connections 2024-01-21T08:21:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2895617895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32380f6960 0x7f32380f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2895617895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f56c0 0x7f32380f5ae0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 -- 172.21.15.46:0/2895617895 >> 172.21.15.46:0/2895617895 conn(0x7f32380f0e20 msgr2=0x7f32380f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 -- 172.21.15.46:0/2895617895 shutdown_connections 2024-01-21T08:21:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.978+0000 7f323e6cc700 1 -- 172.21.15.46:0/2895617895 wait complete. 2024-01-21T08:21:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f323e6cc700 1 Processor -- start 2024-01-21T08:21:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f323e6cc700 1 -- start start 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f323e6cc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32380f56c0 0x7f3238109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f323e6cc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f6960 0x7f323810c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f323e6cc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3238060940 con 0x7f32380f6960 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f323e6cc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3238060a80 con 0x7f32380f56c0 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f32377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f6960 0x7f323810c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f32377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f6960 0x7f323810c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55380/0 (socket says 172.21.15.46:55380) 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.979+0000 7f32377fe700 1 -- 172.21.15.46:0/2134530562 learned_addr learned my addr 172.21.15.46:0/2134530562 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f3237fff700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32380f56c0 0x7f3238109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f32377fe700 1 -- 172.21.15.46:0/2134530562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32380f56c0 msgr2=0x7f3238109dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f32377fe700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32380f56c0 0x7f3238109dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f32377fe700 1 -- 172.21.15.46:0/2134530562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228005040 con 0x7f32380f6960 2024-01-21T08:21:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f32377fe700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f6960 0x7f323810c2d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f322c0026d0 tx=0x7f322c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f3234ff9700 1 -- 172.21.15.46:0/2134530562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c0079c0 con 0x7f32380f6960 2024-01-21T08:21:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323810c880 con 0x7f32380f6960 2024-01-21T08:21:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f3234ff9700 1 -- 172.21.15.46:0/2134530562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f322c00f040 con 0x7f32380f6960 2024-01-21T08:21:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f3234ff9700 1 -- 172.21.15.46:0/2134530562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c0148b0 con 0x7f32380f6960 2024-01-21T08:21:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.980+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323810ce40 con 0x7f32380f6960 2024-01-21T08:21:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.981+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3238045ad0 con 0x7f32380f6960 2024-01-21T08:21:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.981+0000 7f3234ff9700 1 -- 172.21.15.46:0/2134530562 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f322c007b20 con 0x7f32380f6960 2024-01-21T08:21:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.982+0000 7f3234ff9700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f322005c2a0 0x7f322005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.982+0000 7f3234ff9700 1 -- 172.21.15.46:0/2134530562 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f322c075b00 con 0x7f32380f6960 2024-01-21T08:21:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.982+0000 7f3237fff700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f322005c2a0 0x7f322005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.982+0000 7f3237fff700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f322005c2a0 0x7f322005e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f32380605c0 tx=0x7f3228009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:45.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:45.986+0000 7f3234ff9700 1 -- 172.21.15.46:0/2134530562 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f322c048b00 con 0x7f32380f6960 2024-01-21T08:21:46.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.204+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3238027a00 con 0x7f32380f6960 2024-01-21T08:21:46.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.205+0000 7f3234ff9700 1 -- 172.21.15.46:0/2134530562 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f322c04c120 con 0x7f32380f6960 2024-01-21T08:21:46.207 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f322005c2a0 msgr2=0x7f322005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f322005c2a0 0x7f322005e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f32380605c0 tx=0x7f3228009b30).stop 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f6960 msgr2=0x7f323810c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f6960 0x7f323810c2d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f322c0026d0 tx=0x7f322c009de0).stop 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 shutdown_connections 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f322005c2a0 0x7f322005e760 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32380f56c0 0x7f3238109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 --2- 172.21.15.46:0/2134530562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32380f6960 0x7f323810c2d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.206+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 >> 172.21.15.46:0/2134530562 conn(0x7f32380f0e20 msgr2=0x7f32380fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.207+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 shutdown_connections 2024-01-21T08:21:46.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:46.207+0000 7f323e6cc700 1 -- 172.21.15.46:0/2134530562 wait complete. 2024-01-21T08:21:46.588 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:47.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:46.135022+0000 mgr.smithi046.lmxnwc (mgr.14164) 551 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:47.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:46.205705+0000 mon.smithi046 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.46:0/2134530562' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:47.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:46.135022+0000 mgr.smithi046.lmxnwc (mgr.14164) 551 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:47.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:46.205705+0000 mon.smithi046 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.46:0/2134530562' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:49.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:48.135588+0000 mgr.smithi046.lmxnwc (mgr.14164) 552 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:49.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:48.135588+0000 mgr.smithi046.lmxnwc (mgr.14164) 552 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:49.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:49.832 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:50.591 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:50.136064+0000 mgr.smithi046.lmxnwc (mgr.14164) 553 : cluster [DBG] pgmap v521: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:50.136064+0000 mgr.smithi046.lmxnwc (mgr.14164) 553 : cluster [DBG] pgmap v521: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.762+0000 7fefc310b700 1 -- 172.21.15.46:0/668575631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f48e0 msgr2=0x7fefbc0f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.762+0000 7fefc310b700 1 --2- 172.21.15.46:0/668575631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f48e0 0x7fefbc0f4d00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fefb80099e0 tx=0x7fefb8004d10).stop 2024-01-21T08:21:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 -- 172.21.15.46:0/668575631 shutdown_connections 2024-01-21T08:21:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 --2- 172.21.15.46:0/668575631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefbc0f5280 0x7fefbc0f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 --2- 172.21.15.46:0/668575631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f48e0 0x7fefbc0f4d00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 -- 172.21.15.46:0/668575631 >> 172.21.15.46:0/668575631 conn(0x7fefbc0f0400 msgr2=0x7fefbc0f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 -- 172.21.15.46:0/668575631 shutdown_connections 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 -- 172.21.15.46:0/668575631 wait complete. 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 Processor -- start 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 -- start start 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefbc0f48e0 0x7fefbc10d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f5280 0x7fefbc10db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc10e0f0 con 0x7fefbc0f5280 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc310b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc10e230 con 0x7fefbc0f48e0 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc1908700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f5280 0x7fefbc10db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc1908700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f5280 0x7fefbc10db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50208/0 (socket says 172.21.15.46:50208) 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.763+0000 7fefc1908700 1 -- 172.21.15.46:0/3354996534 learned_addr learned my addr 172.21.15.46:0/3354996534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefc2109700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefbc0f48e0 0x7fefbc10d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefc1908700 1 -- 172.21.15.46:0/3354996534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefbc0f48e0 msgr2=0x7fefbc10d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefc1908700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefbc0f48e0 0x7fefbc10d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefc1908700 1 -- 172.21.15.46:0/3354996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb8009640 con 0x7fefbc0f5280 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefc1908700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f5280 0x7fefbc10db40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fefac00c790 tx=0x7fefac007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefb2ffd700 1 -- 172.21.15.46:0/3354996534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefac013630 con 0x7fefbc0f5280 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefb2ffd700 1 -- 172.21.15.46:0/3354996534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefac013c40 con 0x7fefbc0f5280 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc10ebd0 con 0x7fefbc0f5280 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefb2ffd700 1 -- 172.21.15.46:0/3354996534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefac002b80 con 0x7fefbc0f5280 2024-01-21T08:21:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.764+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc10f110 con 0x7fefbc0f5280 2024-01-21T08:21:50.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.765+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc045ad0 con 0x7fefbc0f5280 2024-01-21T08:21:50.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.765+0000 7fefb2ffd700 1 -- 172.21.15.46:0/3354996534 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fefac01a060 con 0x7fefbc0f5280 2024-01-21T08:21:50.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.766+0000 7fefb2ffd700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fefa805c2b0 0x7fefa805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:50.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.766+0000 7fefb2ffd700 1 -- 172.21.15.46:0/3354996534 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fefac075c60 con 0x7fefbc0f5280 2024-01-21T08:21:50.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.766+0000 7fefc2109700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fefa805c2b0 0x7fefa805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:50.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.766+0000 7fefc2109700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fefa805c2b0 0x7fefa805e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fefb800b620 tx=0x7fefb8004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:50.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.770+0000 7fefb2ffd700 1 -- 172.21.15.46:0/3354996534 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefac048ce0 con 0x7fefbc0f5280 2024-01-21T08:21:50.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.989+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fefbc027a00 con 0x7fefbc0f5280 2024-01-21T08:21:50.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.989+0000 7fefb2ffd700 1 -- 172.21.15.46:0/3354996534 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fefac04c300 con 0x7fefbc0f5280 2024-01-21T08:21:50.991 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fefa805c2b0 msgr2=0x7fefa805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fefa805c2b0 0x7fefa805e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fefb800b620 tx=0x7fefb8004d10).stop 2024-01-21T08:21:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f5280 msgr2=0x7fefbc10db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f5280 0x7fefbc10db40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fefac00c790 tx=0x7fefac007d40).stop 2024-01-21T08:21:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 shutdown_connections 2024-01-21T08:21:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fefa805c2b0 0x7fefa805e770 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:50.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefbc0f48e0 0x7fefbc10d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:50.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 --2- 172.21.15.46:0/3354996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefbc0f5280 0x7fefbc10db40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:50.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 >> 172.21.15.46:0/3354996534 conn(0x7fefbc0f0400 msgr2=0x7fefbc0fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:50.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 shutdown_connections 2024-01-21T08:21:50.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:50.991+0000 7fefc310b700 1 -- 172.21.15.46:0/3354996534 wait complete. 2024-01-21T08:21:51.421 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:51.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:51 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:50.989918+0000 mon.smithi046 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.46:0/3354996534' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:51.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:51 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:50.989918+0000 mon.smithi046 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.46:0/3354996534' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:52.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:52 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:52.136445+0000 mgr.smithi046.lmxnwc (mgr.14164) 554 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:52.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:52 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:52.136445+0000 mgr.smithi046.lmxnwc (mgr.14164) 554 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:54.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:55.408 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:54.136904+0000 mgr.smithi046.lmxnwc (mgr.14164) 555 : cluster [DBG] pgmap v523: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:55.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:54.136904+0000 mgr.smithi046.lmxnwc (mgr.14164) 555 : cluster [DBG] pgmap v523: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 -- 172.21.15.46:0/147935266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 msgr2=0x7f05780f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 --2- 172.21.15.46:0/147935266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f05780f6dc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f056c004660 tx=0x7f056c009b30).stop 2024-01-21T08:21:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 -- 172.21.15.46:0/147935266 shutdown_connections 2024-01-21T08:21:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 --2- 172.21.15.46:0/147935266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f05780f6dc0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 --2- 172.21.15.46:0/147935266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05780f56c0 0x7f05780f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 -- 172.21.15.46:0/147935266 >> 172.21.15.46:0/147935266 conn(0x7f05780f0e20 msgr2=0x7f05780f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 -- 172.21.15.46:0/147935266 shutdown_connections 2024-01-21T08:21:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.598+0000 7f057e36b700 1 -- 172.21.15.46:0/147935266 wait complete. 2024-01-21T08:21:55.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f057e36b700 1 Processor -- start 2024-01-21T08:21:55.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f057e36b700 1 -- start start 2024-01-21T08:21:55.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f057e36b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05780f56c0 0x7f0578109de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:55.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f057e36b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f057810c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:55.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f057e36b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578060940 con 0x7f05780f6960 2024-01-21T08:21:55.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f057e36b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578060a80 con 0x7f05780f56c0 2024-01-21T08:21:55.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f05777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f057810c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f05777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f057810c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50220/0 (socket says 172.21.15.46:50220) 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f05777fe700 1 -- 172.21.15.46:0/3284554865 learned_addr learned my addr 172.21.15.46:0/3284554865 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f0577fff700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05780f56c0 0x7f0578109de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f05777fe700 1 -- 172.21.15.46:0/3284554865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05780f56c0 msgr2=0x7f0578109de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f05777fe700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05780f56c0 0x7f0578109de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f05777fe700 1 -- 172.21.15.46:0/3284554865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c005040 con 0x7f05780f6960 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f0577fff700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05780f56c0 0x7f0578109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.599+0000 7f05777fe700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f057810c2e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f056c009830 tx=0x7f056c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.600+0000 7f0574ff9700 1 -- 172.21.15.46:0/3284554865 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056c018070 con 0x7f05780f6960 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.600+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057810c830 con 0x7f05780f6960 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.600+0000 7f0574ff9700 1 -- 172.21.15.46:0/3284554865 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f056c013410 con 0x7f05780f6960 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.600+0000 7f0574ff9700 1 -- 172.21.15.46:0/3284554865 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056c01c620 con 0x7f05780f6960 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.600+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057810cd90 con 0x7f05780f6960 2024-01-21T08:21:55.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.601+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0578045ad0 con 0x7f05780f6960 2024-01-21T08:21:55.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.601+0000 7f0574ff9700 1 -- 172.21.15.46:0/3284554865 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f056c01c890 con 0x7f05780f6960 2024-01-21T08:21:55.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.601+0000 7f0574ff9700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f056005c2a0 0x7f056005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:21:55.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.602+0000 7f0574ff9700 1 -- 172.21.15.46:0/3284554865 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f056c077a70 con 0x7f05780f6960 2024-01-21T08:21:55.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.602+0000 7f0577fff700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f056005c2a0 0x7f056005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:21:55.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.602+0000 7f0577fff700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f056005c2a0 0x7f056005e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f05780605c0 tx=0x7f0568009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:21:55.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.605+0000 7f0574ff9700 1 -- 172.21.15.46:0/3284554865 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f056c04a9d0 con 0x7f05780f6960 2024-01-21T08:21:55.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.831+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0578027a00 con 0x7f05780f6960 2024-01-21T08:21:55.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.831+0000 7f0574ff9700 1 -- 172.21.15.46:0/3284554865 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f056c04dff0 con 0x7f05780f6960 2024-01-21T08:21:55.833 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:21:55.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f056005c2a0 msgr2=0x7f056005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:55.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f056005c2a0 0x7f056005e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f05780605c0 tx=0x7f0568009040).stop 2024-01-21T08:21:55.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 msgr2=0x7f057810c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f057810c2e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f056c009830 tx=0x7f056c009b30).stop 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 shutdown_connections 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f056005c2a0 0x7f056005e760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05780f56c0 0x7f0578109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 --2- 172.21.15.46:0/3284554865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05780f6960 0x7f057810c2e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 >> 172.21.15.46:0/3284554865 conn(0x7f05780f0e20 msgr2=0x7f05780fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 shutdown_connections 2024-01-21T08:21:55.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:21:55.833+0000 7f057e36b700 1 -- 172.21.15.46:0/3284554865 wait complete. 2024-01-21T08:21:56.213 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:21:56.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:56 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:55.831959+0000 mon.smithi046 (mon.0) 558 : audit [DBG] from='client.? 172.21.15.46:0/3284554865' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:56.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:56 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:55.831959+0000 mon.smithi046 (mon.0) 558 : audit [DBG] from='client.? 172.21.15.46:0/3284554865' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:21:57.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:56.137213+0000 mgr.smithi046.lmxnwc (mgr.14164) 556 : cluster [DBG] pgmap v524: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:57.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:56.137213+0000 mgr.smithi046.lmxnwc (mgr.14164) 556 : cluster [DBG] pgmap v524: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:58.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:21:57.279559+0000 mgr.smithi046.lmxnwc (mgr.14164) 557 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2024-01-21T08:21:58.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.424 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:21:58.425 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: yield (conn, connr) 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: code, '\n'.join(err))) 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: Traceback (most recent call last): 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: main() 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:21:58.426 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: r = ctx.func(ctx) 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: return func(ctx) 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: ports=daemon_ports) 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.282105+0000 mon.smithi046 (mon.0) 559 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.282530+0000 mgr.smithi046.lmxnwc (mgr.14164) 558 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.283749+0000 mon.smithi046 (mon.0) 560 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.46:9093"}]: dispatch 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.284048+0000 mgr.smithi046.lmxnwc (mgr.14164) 559 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.46:9093"}]: dispatch 2024-01-21T08:21:58.427 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.288869+0000 mon.smithi046 (mon.0) 561 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.293430+0000 mon.smithi046 (mon.0) 562 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.293822+0000 mgr.smithi046.lmxnwc (mgr.14164) 560 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.299654+0000 mon.smithi046 (mon.0) 563 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.46:3000"}]: dispatch 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.299970+0000 mgr.smithi046.lmxnwc (mgr.14164) 561 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.46:3000"}]: dispatch 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.304312+0000 mon.smithi046 (mon.0) 564 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.306809+0000 mon.smithi046 (mon.0) 565 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.536004+0000 mon.smithi046 (mon.0) 566 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:58.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:57.544154+0000 mon.smithi046 (mon.0) 567 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:21:57.279559+0000 mgr.smithi046.lmxnwc (mgr.14164) 557 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:21:58.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ports=daemon_ports) 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: yield (conn, connr) 2024-01-21T08:21:58.493 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: code, '\n'.join(err))) 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi078 ... 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Non-zero exit code 1 from systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr Job for ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service failed because a timeout was exceeded. 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: systemctl: stderr See "systemctl status ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078.service" and "journalctl -xe" for details. 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: Traceback (most recent call last): 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: main() 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: r = ctx.func(ctx) 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: return func(ctx) 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T08:21:58.494 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: ports=daemon_ports) 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: File "/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: RuntimeError: Failed command: systemctl restart ceph-a17ce476-b833-11ee-95b0-87774f69a715@node-exporter.smithi078 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.282105+0000 mon.smithi046 (mon.0) 559 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.282530+0000 mgr.smithi046.lmxnwc (mgr.14164) 558 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.283749+0000 mon.smithi046 (mon.0) 560 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.46:9093"}]: dispatch 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.284048+0000 mgr.smithi046.lmxnwc (mgr.14164) 559 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.46:9093"}]: dispatch 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.288869+0000 mon.smithi046 (mon.0) 561 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.293430+0000 mon.smithi046 (mon.0) 562 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.293822+0000 mgr.smithi046.lmxnwc (mgr.14164) 560 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-21T08:21:58.495 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.299654+0000 mon.smithi046 (mon.0) 563 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.46:3000"}]: dispatch 2024-01-21T08:21:58.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.299970+0000 mgr.smithi046.lmxnwc (mgr.14164) 561 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.46:3000"}]: dispatch 2024-01-21T08:21:58.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.304312+0000 mon.smithi046 (mon.0) 564 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:58.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.306809+0000 mon.smithi046 (mon.0) 565 : audit [DBG] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T08:21:58.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.536004+0000 mon.smithi046 (mon.0) 566 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:58.496 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:58 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:57.544154+0000 mon.smithi046 (mon.0) 567 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:59.214 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:21:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:21:59.588 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:21:58.137543+0000 mgr.smithi046.lmxnwc (mgr.14164) 562 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:59.630 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:58.428162+0000 mon.smithi046 (mon.0) 568 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:59.630 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:21:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:21:59.274294+0000 mon.smithi046 (mon.0) 569 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:59.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:21:58.137543+0000 mgr.smithi046.lmxnwc (mgr.14164) 562 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:21:59.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:58.428162+0000 mon.smithi046 (mon.0) 568 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:21:59.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:21:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:21:59.274294+0000 mon.smithi046 (mon.0) 569 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 -- 172.21.15.46:0/1522650145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f87e0 msgr2=0x7fa0e40f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/1522650145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f87e0 0x7fa0e40f4960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fa0d8004660 tx=0x7fa0d8009b30).stop 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 -- 172.21.15.46:0/1522650145 shutdown_connections 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/1522650145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f87e0 0x7fa0e40f4960 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/1522650145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0e40f7e40 0x7fa0e40f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 -- 172.21.15.46:0/1522650145 >> 172.21.15.46:0/1522650145 conn(0x7fa0e40f05e0 msgr2=0x7fa0e40f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 -- 172.21.15.46:0/1522650145 shutdown_connections 2024-01-21T08:22:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.671+0000 7fa0e9c35700 1 -- 172.21.15.46:0/1522650145 wait complete. 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e9c35700 1 Processor -- start 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e9c35700 1 -- start start 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e9c35700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f7e40 0x7fa0e4107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e9c35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0e40f87e0 0x7fa0e4107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e9c35700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4107ad0 con 0x7fa0e40f7e40 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e9c35700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4107c10 con 0x7fa0e40f87e0 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f7e40 0x7fa0e4107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f7e40 0x7fa0e4107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47436/0 (socket says 172.21.15.46:47436) 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e37fe700 1 -- 172.21.15.46:0/3918222542 learned_addr learned my addr 172.21.15.46:0/3918222542 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e2ffd700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0e40f87e0 0x7fa0e4107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e37fe700 1 -- 172.21.15.46:0/3918222542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0e40f87e0 msgr2=0x7fa0e4107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e37fe700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0e40f87e0 0x7fa0e4107520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e37fe700 1 -- 172.21.15.46:0/3918222542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d8005040 con 0x7fa0e40f7e40 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.672+0000 7fa0e37fe700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f7e40 0x7fa0e4107030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa0d4002700 tx=0x7fa0d4009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.673+0000 7fa0cbfff700 1 -- 172.21.15.46:0/3918222542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d40079c0 con 0x7fa0e40f7e40 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.673+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e41085b0 con 0x7fa0e40f7e40 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.673+0000 7fa0cbfff700 1 -- 172.21.15.46:0/3918222542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0d400f040 con 0x7fa0e40f7e40 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.673+0000 7fa0cbfff700 1 -- 172.21.15.46:0/3918222542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d4014850 con 0x7fa0e40f7e40 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.673+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e4108b70 con 0x7fa0e40f7e40 2024-01-21T08:22:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.674+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e4045ad0 con 0x7fa0e40f7e40 2024-01-21T08:22:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.674+0000 7fa0cbfff700 1 -- 172.21.15.46:0/3918222542 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa0d401b060 con 0x7fa0e40f7e40 2024-01-21T08:22:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.674+0000 7fa0cbfff700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa0cc05c2a0 0x7fa0cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.675+0000 7fa0cbfff700 1 -- 172.21.15.46:0/3918222542 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa0d4075aa0 con 0x7fa0e40f7e40 2024-01-21T08:22:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.675+0000 7fa0e2ffd700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa0cc05c2a0 0x7fa0cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.675+0000 7fa0e2ffd700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa0cc05c2a0 0x7fa0cc05e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fa0d8009f50 tx=0x7fa0d8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.679+0000 7fa0cbfff700 1 -- 172.21.15.46:0/3918222542 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0d4048aa0 con 0x7fa0e40f7e40 2024-01-21T08:22:00.731 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:00.137998+0000 mgr.smithi046.lmxnwc (mgr.14164) 563 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:00.137998+0000 mgr.smithi046.lmxnwc (mgr.14164) 563 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:00.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.905+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa0e40ff100 con 0x7fa0e40f7e40 2024-01-21T08:22:00.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.906+0000 7fa0cbfff700 1 -- 172.21.15.46:0/3918222542 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fa0d404c0c0 con 0x7fa0e40f7e40 2024-01-21T08:22:00.908 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:00.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa0cc05c2a0 msgr2=0x7fa0cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:00.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa0cc05c2a0 0x7fa0cc05e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fa0d8009f50 tx=0x7fa0d8009b30).stop 2024-01-21T08:22:00.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f7e40 msgr2=0x7fa0e4107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:00.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f7e40 0x7fa0e4107030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa0d4002700 tx=0x7fa0d4009e00).stop 2024-01-21T08:22:00.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 shutdown_connections 2024-01-21T08:22:00.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fa0cc05c2a0 0x7fa0cc05e760 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:00.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e40f7e40 0x7fa0e4107030 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:00.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 --2- 172.21.15.46:0/3918222542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0e40f87e0 0x7fa0e4107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:00.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 >> 172.21.15.46:0/3918222542 conn(0x7fa0e40f05e0 msgr2=0x7fa0e40fb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:00.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 shutdown_connections 2024-01-21T08:22:00.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:00.908+0000 7fa0e9c35700 1 -- 172.21.15.46:0/3918222542 wait complete. 2024-01-21T08:22:01.286 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:01.893 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:22:00.730945+0000 mgr.smithi046.lmxnwc (mgr.14164) 564 : cephadm [INF] Detected new or changed devices on smithi078 2024-01-21T08:22:01.893 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:00.735837+0000 mon.smithi046 (mon.0) 570 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:22:01.893 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:00.906832+0000 mon.smithi046 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.46:0/3918222542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:01.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:22:00.730945+0000 mgr.smithi046.lmxnwc (mgr.14164) 564 : cephadm [INF] Detected new or changed devices on smithi078 2024-01-21T08:22:01.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:00.735837+0000 mon.smithi046 (mon.0) 570 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:22:01.992 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:01 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:00.906832+0000 mon.smithi046 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.46:0/3918222542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:03.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:02.138328+0000 mgr.smithi046.lmxnwc (mgr.14164) 565 : cluster [DBG] pgmap v527: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:03.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:02.197626+0000 mon.smithi046 (mon.0) 572 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:22:03.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:02.442343+0000 mon.smithi046 (mon.0) 573 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:22:03.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cephadm 2024-01-21T08:22:02.446287+0000 mgr.smithi046.lmxnwc (mgr.14164) 566 : cephadm [INF] Deploying daemon prometheus.smithi078 on smithi078 2024-01-21T08:22:03.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:02.138328+0000 mgr.smithi046.lmxnwc (mgr.14164) 565 : cluster [DBG] pgmap v527: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:03.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:02.197626+0000 mon.smithi046 (mon.0) 572 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:22:03.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:02.442343+0000 mon.smithi046 (mon.0) 573 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' 2024-01-21T08:22:03.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cephadm 2024-01-21T08:22:02.446287+0000 mgr.smithi046.lmxnwc (mgr.14164) 566 : cephadm [INF] Deploying daemon prometheus.smithi078 on smithi078 2024-01-21T08:22:04.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:04.533 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:05.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:04.138658+0000 mgr.smithi046.lmxnwc (mgr.14164) 567 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:05.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:04.138658+0000 mgr.smithi046.lmxnwc (mgr.14164) 567 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3303610769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f67d0 msgr2=0x7f4eac0f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3303610769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f67d0 0x7f4eac0f6bf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f4e9c0099e0 tx=0x7f4e9c004d10).stop 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3303610769 shutdown_connections 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3303610769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4eac0f7a70 0x7f4eac0f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3303610769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f67d0 0x7f4eac0f6bf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3303610769 >> 172.21.15.46:0/3303610769 conn(0x7f4eac0f1f50 msgr2=0x7f4eac0f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3303610769 shutdown_connections 2024-01-21T08:22:05.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.532+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3303610769 wait complete. 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4eb0be0700 1 Processor -- start 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4eb0be0700 1 -- start start 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4eb0be0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4eac0f67d0 0x7f4eac1071b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4eb0be0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f7a70 0x7f4eac1076a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4eb0be0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac107c50 con 0x7f4eac0f7a70 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4eb0be0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac107d90 con 0x7f4eac0f67d0 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4ea9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f7a70 0x7f4eac1076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4ea9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f7a70 0x7f4eac1076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47444/0 (socket says 172.21.15.46:47444) 2024-01-21T08:22:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4ea9d9b700 1 -- 172.21.15.46:0/3113771025 learned_addr learned my addr 172.21.15.46:0/3113771025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4eaa59c700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4eac0f67d0 0x7f4eac1071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4ea9d9b700 1 -- 172.21.15.46:0/3113771025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4eac0f67d0 msgr2=0x7f4eac1071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.533+0000 7f4ea9d9b700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4eac0f67d0 0x7f4eac1071b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.534+0000 7f4ea9d9b700 1 -- 172.21.15.46:0/3113771025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c009640 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.534+0000 7f4ea9d9b700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f7a70 0x7f4eac1076a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f4e9400c530 tx=0x7f4e94007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.534+0000 7f4ea2ffd700 1 -- 172.21.15.46:0/3113771025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9400f650 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.534+0000 7f4ea2ffd700 1 -- 172.21.15.46:0/3113771025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e9400fc60 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.534+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac108730 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.534+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac108c70 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.534+0000 7f4ea2ffd700 1 -- 172.21.15.46:0/3113771025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e94012c30 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.535+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac045ad0 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.535+0000 7f4ea2ffd700 1 -- 172.21.15.46:0/3113771025 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4e94010070 con 0x7f4eac0f7a70 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.535+0000 7f4ea2ffd700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e9805c2a0 0x7f4e9805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.536+0000 7f4ea2ffd700 1 -- 172.21.15.46:0/3113771025 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4e940757f0 con 0x7f4eac0f7a70 2024-01-21T08:22:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.536+0000 7f4eaa59c700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e9805c2a0 0x7f4e9805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.536+0000 7f4eaa59c700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e9805c2a0 0x7f4e9805e760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f4e9c00b620 tx=0x7f4e9c004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.539+0000 7f4ea2ffd700 1 -- 172.21.15.46:0/3113771025 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e940452b0 con 0x7f4eac0f7a70 2024-01-21T08:22:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.774+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4eac027a00 con 0x7f4eac0f7a70 2024-01-21T08:22:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.774+0000 7f4ea2ffd700 1 -- 172.21.15.46:0/3113771025 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f4e940487f0 con 0x7f4eac0f7a70 2024-01-21T08:22:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e9805c2a0 msgr2=0x7f4e9805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e9805c2a0 0x7f4e9805e760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f4e9c00b620 tx=0x7f4e9c004d10).stop 2024-01-21T08:22:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f7a70 msgr2=0x7f4eac1076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f7a70 0x7f4eac1076a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f4e9400c530 tx=0x7f4e94007e20).stop 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 shutdown_connections 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f4e9805c2a0 0x7f4e9805e760 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4eac0f67d0 0x7f4eac1071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 --2- 172.21.15.46:0/3113771025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eac0f7a70 0x7f4eac1076a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 >> 172.21.15.46:0/3113771025 conn(0x7f4eac0f1f50 msgr2=0x7f4eac0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 shutdown_connections 2024-01-21T08:22:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:05.776+0000 7f4eb0be0700 1 -- 172.21.15.46:0/3113771025 wait complete. 2024-01-21T08:22:06.169 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:06.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:06 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:05.774975+0000 mon.smithi046 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.46:0/3113771025' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:06.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:05.774975+0000 mon.smithi046 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.46:0/3113771025' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:07.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:06.139025+0000 mgr.smithi046.lmxnwc (mgr.14164) 568 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:07.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:06.139025+0000 mgr.smithi046.lmxnwc (mgr.14164) 568 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:09.170 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:09.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:08.139539+0000 mgr.smithi046.lmxnwc (mgr.14164) 569 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:09.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:08.139539+0000 mgr.smithi046.lmxnwc (mgr.14164) 569 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 -- 172.21.15.46:0/4283032544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 msgr2=0x7fd9400fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 --2- 172.21.15.46:0/4283032544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400fbd70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fd934004660 tx=0x7fd934009b30).stop 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 -- 172.21.15.46:0/4283032544 shutdown_connections 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 --2- 172.21.15.46:0/4283032544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400fbd70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 --2- 172.21.15.46:0/4283032544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd940060870 0x7fd9400f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 -- 172.21.15.46:0/4283032544 >> 172.21.15.46:0/4283032544 conn(0x7fd9400f0e20 msgr2=0x7fd9400f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 -- 172.21.15.46:0/4283032544 shutdown_connections 2024-01-21T08:22:10.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.414+0000 7fd945b7e700 1 -- 172.21.15.46:0/4283032544 wait complete. 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd945b7e700 1 Processor -- start 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd945b7e700 1 -- start start 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd945b7e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd940060870 0x7fd9400f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd945b7e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd945b7e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9400f5800 con 0x7fd9400f9920 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd945b7e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9400f5940 con 0x7fd940060870 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd93effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd93effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47464/0 (socket says 172.21.15.46:47464) 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd93effd700 1 -- 172.21.15.46:0/2917381320 learned_addr learned my addr 172.21.15.46:0/2917381320 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.415+0000 7fd93f7fe700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd940060870 0x7fd9400f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd93effd700 1 -- 172.21.15.46:0/2917381320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd940060870 msgr2=0x7fd9400f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd93effd700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd940060870 0x7fd9400f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd93effd700 1 -- 172.21.15.46:0/2917381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd934005040 con 0x7fd9400f9920 2024-01-21T08:22:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd93effd700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400f52a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fd934009830 tx=0x7fd934009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:10.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd927fff700 1 -- 172.21.15.46:0/2917381320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd934018070 con 0x7fd9400f9920 2024-01-21T08:22:10.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9400f5bc0 con 0x7fd9400f9920 2024-01-21T08:22:10.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd927fff700 1 -- 172.21.15.46:0/2917381320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd934007ba0 con 0x7fd9400f9920 2024-01-21T08:22:10.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd927fff700 1 -- 172.21.15.46:0/2917381320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd93401de20 con 0x7fd9400f9920 2024-01-21T08:22:10.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.416+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9400f6120 con 0x7fd9400f9920 2024-01-21T08:22:10.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.417+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd940045ad0 con 0x7fd9400f9920 2024-01-21T08:22:10.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.417+0000 7fd927fff700 1 -- 172.21.15.46:0/2917381320 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd934007d10 con 0x7fd9400f9920 2024-01-21T08:22:10.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.418+0000 7fd927fff700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd92805c2a0 0x7fd92805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:10.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.418+0000 7fd927fff700 1 -- 172.21.15.46:0/2917381320 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd934077710 con 0x7fd9400f9920 2024-01-21T08:22:10.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.418+0000 7fd93f7fe700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd92805c2a0 0x7fd92805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:10.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.418+0000 7fd93f7fe700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd92805c2a0 0x7fd92805e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fd9400f6bb0 tx=0x7fd930009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:10.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.422+0000 7fd927fff700 1 -- 172.21.15.46:0/2917381320 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd93404a710 con 0x7fd9400f9920 2024-01-21T08:22:10.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.643+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd940027a00 con 0x7fd9400f9920 2024-01-21T08:22:10.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.644+0000 7fd927fff700 1 -- 172.21.15.46:0/2917381320 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fd93404dd30 con 0x7fd9400f9920 2024-01-21T08:22:10.645 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd92805c2a0 msgr2=0x7fd92805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd92805c2a0 0x7fd92805e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fd9400f6bb0 tx=0x7fd930009040).stop 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 msgr2=0x7fd9400f52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400f52a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fd934009830 tx=0x7fd934009b30).stop 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 shutdown_connections 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fd92805c2a0 0x7fd92805e760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd940060870 0x7fd9400f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 --2- 172.21.15.46:0/2917381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9400f9920 0x7fd9400f52a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 >> 172.21.15.46:0/2917381320 conn(0x7fd9400f0e20 msgr2=0x7fd9400f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 shutdown_connections 2024-01-21T08:22:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:10.645+0000 7fd945b7e700 1 -- 172.21.15.46:0/2917381320 wait complete. 2024-01-21T08:22:10.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:10.140063+0000 mgr.smithi046.lmxnwc (mgr.14164) 570 : cluster [DBG] pgmap v531: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:10.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:10.140063+0000 mgr.smithi046.lmxnwc (mgr.14164) 570 : cluster [DBG] pgmap v531: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:11.035 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:11.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:10.644544+0000 mon.smithi046 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.46:0/2917381320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:11.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:10.644544+0000 mon.smithi046 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.46:0/2917381320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:12.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:12 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:12.140589+0000 mgr.smithi046.lmxnwc (mgr.14164) 571 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:12.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:12 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:12.140589+0000 mgr.smithi046.lmxnwc (mgr.14164) 571 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:14.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:15.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:14.140985+0000 mgr.smithi046.lmxnwc (mgr.14164) 572 : cluster [DBG] pgmap v533: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 -- 172.21.15.46:0/3684596836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 msgr2=0x7f3f800f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 --2- 172.21.15.46:0/3684596836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f800f6dc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f3f70004660 tx=0x7f3f70009b30).stop 2024-01-21T08:22:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 -- 172.21.15.46:0/3684596836 shutdown_connections 2024-01-21T08:22:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 --2- 172.21.15.46:0/3684596836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f800f6dc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 --2- 172.21.15.46:0/3684596836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f800f56c0 0x7f3f800f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 -- 172.21.15.46:0/3684596836 >> 172.21.15.46:0/3684596836 conn(0x7f3f800f0e20 msgr2=0x7f3f800f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 -- 172.21.15.46:0/3684596836 shutdown_connections 2024-01-21T08:22:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.245+0000 7f3f859fb700 1 -- 172.21.15.46:0/3684596836 wait complete. 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f859fb700 1 Processor -- start 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f859fb700 1 -- start start 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f859fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f800f56c0 0x7f3f80105fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f859fb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f801064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f859fb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f80106a80 con 0x7f3f800f6960 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f859fb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f80106bc0 con 0x7f3f800f56c0 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f801064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f801064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49026/0 (socket says 172.21.15.46:49026) 2024-01-21T08:22:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7e7fc700 1 -- 172.21.15.46:0/2495957089 learned_addr learned my addr 172.21.15.46:0/2495957089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7effd700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f800f56c0 0x7f3f80105fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7e7fc700 1 -- 172.21.15.46:0/2495957089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f800f56c0 msgr2=0x7f3f80105fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7e7fc700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f800f56c0 0x7f3f80105fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7e7fc700 1 -- 172.21.15.46:0/2495957089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70005040 con 0x7f3f800f6960 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f7e7fc700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f801064d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f3f70009830 tx=0x7f3f70009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f677fe700 1 -- 172.21.15.46:0/2495957089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f70018070 con 0x7f3f800f6960 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f677fe700 1 -- 172.21.15.46:0/2495957089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f70013410 con 0x7f3f800f6960 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f80107500 con 0x7f3f800f6960 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.246+0000 7f3f677fe700 1 -- 172.21.15.46:0/2495957089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f7001c620 con 0x7f3f800f6960 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.247+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f80107a60 con 0x7f3f800f6960 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.247+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f80045ad0 con 0x7f3f800f6960 2024-01-21T08:22:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.248+0000 7f3f677fe700 1 -- 172.21.15.46:0/2495957089 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3f7001c890 con 0x7f3f800f6960 2024-01-21T08:22:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.248+0000 7f3f677fe700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f6805c2a0 0x7f3f6805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.248+0000 7f3f677fe700 1 -- 172.21.15.46:0/2495957089 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3f700779b0 con 0x7f3f800f6960 2024-01-21T08:22:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.248+0000 7f3f7effd700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f6805c2a0 0x7f3f6805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.249+0000 7f3f7effd700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f6805c2a0 0x7f3f6805e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f3f74007fd0 tx=0x7f3f74009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.252+0000 7f3f677fe700 1 -- 172.21.15.46:0/2495957089 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f7004a9d0 con 0x7f3f800f6960 2024-01-21T08:22:15.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.471+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3f801081b0 con 0x7f3f800f6960 2024-01-21T08:22:15.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.472+0000 7f3f677fe700 1 -- 172.21.15.46:0/2495957089 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f3f7004dff0 con 0x7f3f800f6960 2024-01-21T08:22:15.473 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.473+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f6805c2a0 msgr2=0x7f3f6805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.473+0000 7f3f859fb700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f6805c2a0 0x7f3f6805e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f3f74007fd0 tx=0x7f3f74009040).stop 2024-01-21T08:22:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.473+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 msgr2=0x7f3f801064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.473+0000 7f3f859fb700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f801064d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f3f70009830 tx=0x7f3f70009b30).stop 2024-01-21T08:22:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.474+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 shutdown_connections 2024-01-21T08:22:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.474+0000 7f3f859fb700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f3f6805c2a0 0x7f3f6805e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.474+0000 7f3f859fb700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f800f56c0 0x7f3f80105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:15.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.474+0000 7f3f859fb700 1 --2- 172.21.15.46:0/2495957089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f800f6960 0x7f3f801064d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:15.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.474+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 >> 172.21.15.46:0/2495957089 conn(0x7f3f800f0e20 msgr2=0x7f3f800fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:15.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.474+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 shutdown_connections 2024-01-21T08:22:15.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:15.474+0000 7f3f859fb700 1 -- 172.21.15.46:0/2495957089 wait complete. 2024-01-21T08:22:15.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:14.140985+0000 mgr.smithi046.lmxnwc (mgr.14164) 572 : cluster [DBG] pgmap v533: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:15.851 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:16.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:16 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:15.472685+0000 mon.smithi046 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.46:0/2495957089' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:16.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:16 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:15.472685+0000 mon.smithi046 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.46:0/2495957089' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:17.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:16.141434+0000 mgr.smithi046.lmxnwc (mgr.14164) 573 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:17.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:16.141434+0000 mgr.smithi046.lmxnwc (mgr.14164) 573 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:18.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:19.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:18.141862+0000 mgr.smithi046.lmxnwc (mgr.14164) 574 : cluster [DBG] pgmap v535: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:19.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:18.141862+0000 mgr.smithi046.lmxnwc (mgr.14164) 574 : cluster [DBG] pgmap v535: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 -- 172.21.15.46:0/3535854981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 msgr2=0x7f5fa40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 --2- 172.21.15.46:0/3535854981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f5ae0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f5f94004660 tx=0x7f5f94009b30).stop 2024-01-21T08:22:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 -- 172.21.15.46:0/3535854981 shutdown_connections 2024-01-21T08:22:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 --2- 172.21.15.46:0/3535854981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fa40f6960 0x7f5fa40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 --2- 172.21.15.46:0/3535854981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f5ae0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 -- 172.21.15.46:0/3535854981 >> 172.21.15.46:0/3535854981 conn(0x7f5fa40f0e20 msgr2=0x7f5fa40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 -- 172.21.15.46:0/3535854981 shutdown_connections 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 -- 172.21.15.46:0/3535854981 wait complete. 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.288+0000 7f5faa2db700 1 Processor -- start 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5faa2db700 1 -- start start 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5faa2db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5faa2db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fa40f6960 0x7f5fa40f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5faa2db700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa40f8290 con 0x7f5fa40f56c0 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5faa2db700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa40f83d0 con 0x7f5fa40f6960 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49032/0 (socket says 172.21.15.46:49032) 2024-01-21T08:22:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa3fff700 1 -- 172.21.15.46:0/233164151 learned_addr learned my addr 172.21.15.46:0/233164151 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa37fe700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fa40f6960 0x7f5fa40f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa3fff700 1 -- 172.21.15.46:0/233164151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fa40f6960 msgr2=0x7f5fa40f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa3fff700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fa40f6960 0x7f5fa40f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa3fff700 1 -- 172.21.15.46:0/233164151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f94005040 con 0x7f5fa40f56c0 2024-01-21T08:22:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa3fff700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f9820 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f5f94009f50 tx=0x7f5f94009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa0ff9700 1 -- 172.21.15.46:0/233164151 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f94018070 con 0x7f5fa40f56c0 2024-01-21T08:22:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa40f8650 con 0x7f5fa40f56c0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa0ff9700 1 -- 172.21.15.46:0/233164151 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f94013410 con 0x7f5fa40f56c0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa410f110 con 0x7f5fa40f56c0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.289+0000 7f5fa0ff9700 1 -- 172.21.15.46:0/233164151 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f9401c620 con 0x7f5fa40f56c0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.290+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa4045ad0 con 0x7f5fa40f56c0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.291+0000 7f5fa0ff9700 1 -- 172.21.15.46:0/233164151 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5f9401d420 con 0x7f5fa40f56c0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.291+0000 7f5fa0ff9700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5f8c05c2a0 0x7f5f8c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.291+0000 7f5fa0ff9700 1 -- 172.21.15.46:0/233164151 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5f94077a00 con 0x7f5fa40f56c0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.291+0000 7f5fa37fe700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5f8c05c2a0 0x7f5f8c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.292+0000 7f5fa37fe700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5f8c05c2a0 0x7f5f8c05e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f5f98006fd0 tx=0x7f5f98008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:20.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.295+0000 7f5fa0ff9700 1 -- 172.21.15.46:0/233164151 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f9404aa00 con 0x7f5fa40f56c0 2024-01-21T08:22:20.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.518+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5fa40fded0 con 0x7f5fa40f56c0 2024-01-21T08:22:20.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.518+0000 7f5fa0ff9700 1 -- 172.21.15.46:0/233164151 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f5f9404e020 con 0x7f5fa40f56c0 2024-01-21T08:22:20.520 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:20.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.519+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5f8c05c2a0 msgr2=0x7f5f8c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:20.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.519+0000 7f5faa2db700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5f8c05c2a0 0x7f5f8c05e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f5f98006fd0 tx=0x7f5f98008040).stop 2024-01-21T08:22:20.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 msgr2=0x7f5fa40f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:20.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f9820 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f5f94009f50 tx=0x7f5f94009b30).stop 2024-01-21T08:22:20.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 shutdown_connections 2024-01-21T08:22:20.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f5f8c05c2a0 0x7f5f8c05e760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:20.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa40f56c0 0x7f5fa40f9820 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:20.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 --2- 172.21.15.46:0/233164151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fa40f6960 0x7f5fa40f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:20.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 >> 172.21.15.46:0/233164151 conn(0x7f5fa40f0e20 msgr2=0x7f5fa40603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:20.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 shutdown_connections 2024-01-21T08:22:20.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:20.520+0000 7f5faa2db700 1 -- 172.21.15.46:0/233164151 wait complete. 2024-01-21T08:22:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:20.142248+0000 mgr.smithi046.lmxnwc (mgr.14164) 575 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:20.142248+0000 mgr.smithi046.lmxnwc (mgr.14164) 575 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:21.000 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:20.518880+0000 mon.smithi046 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.46:0/233164151' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:20.764562+0000 mon.smithi046 (mon.0) 578 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:22:21.741 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:20.765378+0000 mon.smithi046 (mon.0) 579 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:22:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:20.518880+0000 mon.smithi046 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.46:0/233164151' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:20.764562+0000 mon.smithi046 (mon.0) 578 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:22:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:20.765378+0000 mon.smithi046 (mon.0) 579 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:22:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:22.142655+0000 mgr.smithi046.lmxnwc (mgr.14164) 576 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:22.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:22.142655+0000 mgr.smithi046.lmxnwc (mgr.14164) 576 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:24.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:24.249 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:25.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 -- 172.21.15.46:0/3290725041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 msgr2=0x7f01b00f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:25.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3290725041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b00f6be0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f01a0004660 tx=0x7f01a0009b30).stop 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 -- 172.21.15.46:0/3290725041 shutdown_connections 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3290725041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01b00f7a60 0x7f01b00f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3290725041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b00f6be0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 -- 172.21.15.46:0/3290725041 >> 172.21.15.46:0/3290725041 conn(0x7f01b00f1f20 msgr2=0x7f01b00f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 -- 172.21.15.46:0/3290725041 shutdown_connections 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 -- 172.21.15.46:0/3290725041 wait complete. 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 Processor -- start 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.209+0000 7f01b80bd700 1 -- start start 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b80bd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b01070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b80bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01b00f7a60 0x7f01b0107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b80bd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0107b40 con 0x7f01b00f67c0 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b80bd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0107c80 con 0x7f01b00f7a60 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5e59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b01070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5e59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b01070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32846/0 (socket says 172.21.15.46:32846) 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5e59700 1 -- 172.21.15.46:0/3277036640 learned_addr learned my addr 172.21.15.46:0/3277036640 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5658700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01b00f7a60 0x7f01b0107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5e59700 1 -- 172.21.15.46:0/3277036640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01b00f7a60 msgr2=0x7f01b0107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5e59700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01b00f7a60 0x7f01b0107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5e59700 1 -- 172.21.15.46:0/3277036640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0005040 con 0x7f01b00f67c0 2024-01-21T08:22:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b5e59700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b01070a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f01a0004e30 tx=0x7f01a0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01a67fc700 1 -- 172.21.15.46:0/3277036640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01a0018070 con 0x7f01b00f67c0 2024-01-21T08:22:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b01085c0 con 0x7f01b00f67c0 2024-01-21T08:22:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01a67fc700 1 -- 172.21.15.46:0/3277036640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01a0013410 con 0x7f01b00f67c0 2024-01-21T08:22:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01a67fc700 1 -- 172.21.15.46:0/3277036640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01a001de20 con 0x7f01b00f67c0 2024-01-21T08:22:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.210+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b0108b20 con 0x7f01b00f67c0 2024-01-21T08:22:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.211+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b0045ad0 con 0x7f01b00f67c0 2024-01-21T08:22:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.212+0000 7f01a67fc700 1 -- 172.21.15.46:0/3277036640 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f01a001d420 con 0x7f01b00f67c0 2024-01-21T08:22:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.212+0000 7f01a67fc700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f019c05c2a0 0x7f019c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.212+0000 7f01a67fc700 1 -- 172.21.15.46:0/3277036640 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f01a0077800 con 0x7f01b00f67c0 2024-01-21T08:22:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.212+0000 7f01b5658700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f019c05c2a0 0x7f019c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.213+0000 7f01b5658700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f019c05c2a0 0x7f019c05e760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f01ac004c40 tx=0x7f01ac00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.216+0000 7f01a67fc700 1 -- 172.21.15.46:0/3277036640 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01a004a800 con 0x7f01b00f67c0 2024-01-21T08:22:25.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.435+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01b0027a00 con 0x7f01b00f67c0 2024-01-21T08:22:25.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.435+0000 7f01a67fc700 1 -- 172.21.15.46:0/3277036640 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f01a004de20 con 0x7f01b00f67c0 2024-01-21T08:22:25.437 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f019c05c2a0 msgr2=0x7f019c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f019c05c2a0 0x7f019c05e760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f01ac004c40 tx=0x7f01ac00b040).stop 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 msgr2=0x7f01b01070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b01070a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f01a0004e30 tx=0x7f01a0009b30).stop 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 shutdown_connections 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f019c05c2a0 0x7f019c05e760 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00f67c0 0x7f01b01070a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 --2- 172.21.15.46:0/3277036640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01b00f7a60 0x7f01b0107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 >> 172.21.15.46:0/3277036640 conn(0x7f01b00f1f20 msgr2=0x7f01b00fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 shutdown_connections 2024-01-21T08:22:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:25.437+0000 7f01b80bd700 1 -- 172.21.15.46:0/3277036640 wait complete. 2024-01-21T08:22:25.460 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:25 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:24.143030+0000 mgr.smithi046.lmxnwc (mgr.14164) 577 : cluster [DBG] pgmap v538: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:25.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:25 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:24.143030+0000 mgr.smithi046.lmxnwc (mgr.14164) 577 : cluster [DBG] pgmap v538: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:25.824 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:26.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:26 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:25.436236+0000 mon.smithi046 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.46:0/3277036640' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:26.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:26 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:25.436236+0000 mon.smithi046 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.46:0/3277036640' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:27.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:26.143461+0000 mgr.smithi046.lmxnwc (mgr.14164) 578 : cluster [DBG] pgmap v539: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:27.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:26.143461+0000 mgr.smithi046.lmxnwc (mgr.14164) 578 : cluster [DBG] pgmap v539: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:28.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:29.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:28.143943+0000 mgr.smithi046.lmxnwc (mgr.14164) 579 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:29.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:28.143943+0000 mgr.smithi046.lmxnwc (mgr.14164) 579 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 -- 172.21.15.46:0/1539726289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 msgr2=0x7f63440f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1539726289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f63440f7ee0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f6338004660 tx=0x7f6338009b30).stop 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 -- 172.21.15.46:0/1539726289 shutdown_connections 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1539726289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f63440f7ee0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1539726289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63440f67e0 0x7f63440f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 -- 172.21.15.46:0/1539726289 >> 172.21.15.46:0/1539726289 conn(0x7f63440f1f80 msgr2=0x7f63440f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 -- 172.21.15.46:0/1539726289 shutdown_connections 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.069+0000 7f634b6dd700 1 -- 172.21.15.46:0/1539726289 wait complete. 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f634b6dd700 1 Processor -- start 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f634b6dd700 1 -- start start 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f634b6dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63440f67e0 0x7f634406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f634b6dd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f6344069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f634b6dd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634406a380 con 0x7f63440f7a80 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f634b6dd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634406a4c0 con 0x7f63440f67e0 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f6348c78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f6344069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f6348c78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f6344069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32870/0 (socket says 172.21.15.46:32870) 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f6348c78700 1 -- 172.21.15.46:0/1914610410 learned_addr learned my addr 172.21.15.46:0/1914610410 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f6349479700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63440f67e0 0x7f634406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f6348c78700 1 -- 172.21.15.46:0/1914610410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63440f67e0 msgr2=0x7f634406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f6348c78700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63440f67e0 0x7f634406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.070+0000 7f6348c78700 1 -- 172.21.15.46:0/1914610410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6338005040 con 0x7f63440f7a80 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.071+0000 7f6348c78700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f6344069e20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f6338009830 tx=0x7f6338009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.071+0000 7f6335ffb700 1 -- 172.21.15.46:0/1914610410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6338018070 con 0x7f63440f7a80 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.071+0000 7f6335ffb700 1 -- 172.21.15.46:0/1914610410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6338013410 con 0x7f63440f7a80 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.071+0000 7f6335ffb700 1 -- 172.21.15.46:0/1914610410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f633801c400 con 0x7f63440f7a80 2024-01-21T08:22:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.071+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634406a740 con 0x7f63440f7a80 2024-01-21T08:22:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.071+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634406aca0 con 0x7f63440f7a80 2024-01-21T08:22:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.072+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6344045ad0 con 0x7f63440f7a80 2024-01-21T08:22:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.073+0000 7f6335ffb700 1 -- 172.21.15.46:0/1914610410 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f633801d420 con 0x7f63440f7a80 2024-01-21T08:22:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.073+0000 7f6335ffb700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f633005c2a0 0x7f633005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.073+0000 7f6335ffb700 1 -- 172.21.15.46:0/1914610410 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6338077920 con 0x7f63440f7a80 2024-01-21T08:22:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.073+0000 7f6349479700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f633005c2a0 0x7f633005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.073+0000 7f6349479700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f633005c2a0 0x7f633005e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f634406b730 tx=0x7f6340009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:30.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.077+0000 7f6335ffb700 1 -- 172.21.15.46:0/1914610410 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f633804a940 con 0x7f63440f7a80 2024-01-21T08:22:30.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.296+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6344027a00 con 0x7f63440f7a80 2024-01-21T08:22:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.297+0000 7f6335ffb700 1 -- 172.21.15.46:0/1914610410 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f633804df60 con 0x7f63440f7a80 2024-01-21T08:22:30.299 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.298+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f633005c2a0 msgr2=0x7f633005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.298+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f633005c2a0 0x7f633005e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f634406b730 tx=0x7f6340009040).stop 2024-01-21T08:22:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.298+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 msgr2=0x7f6344069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.298+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f6344069e20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f6338009830 tx=0x7f6338009b30).stop 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.298+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 shutdown_connections 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.298+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f633005c2a0 0x7f633005e760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.299+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63440f67e0 0x7f634406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.299+0000 7f634b6dd700 1 --2- 172.21.15.46:0/1914610410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63440f7a80 0x7f6344069e20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.299+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 >> 172.21.15.46:0/1914610410 conn(0x7f63440f1f80 msgr2=0x7f63440fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.299+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 shutdown_connections 2024-01-21T08:22:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:30.299+0000 7f634b6dd700 1 -- 172.21.15.46:0/1914610410 wait complete. 2024-01-21T08:22:30.664 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:30.144247+0000 mgr.smithi046.lmxnwc (mgr.14164) 580 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:30.664 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:30.297740+0000 mon.smithi046 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.46:0/1914610410' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:30.665 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:30.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:30.144247+0000 mgr.smithi046.lmxnwc (mgr.14164) 580 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:30.742 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:30.297740+0000 mon.smithi046 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.46:0/1914610410' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:33.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:32.144667+0000 mgr.smithi046.lmxnwc (mgr.14164) 581 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:33.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:32.144667+0000 mgr.smithi046.lmxnwc (mgr.14164) 581 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:33.666 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:33.911 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:35.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 -- 172.21.15.46:0/271367116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 msgr2=0x7fee100f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 --2- 172.21.15.46:0/271367116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee100f5b00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7fedfc004660 tx=0x7fedfc009b30).stop 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 -- 172.21.15.46:0/271367116 shutdown_connections 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 --2- 172.21.15.46:0/271367116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee100f6980 0x7fee100f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 --2- 172.21.15.46:0/271367116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee100f5b00 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 -- 172.21.15.46:0/271367116 >> 172.21.15.46:0/271367116 conn(0x7fee100f0e20 msgr2=0x7fee100f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 -- 172.21.15.46:0/271367116 shutdown_connections 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 -- 172.21.15.46:0/271367116 wait complete. 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.016+0000 7fee14e28700 1 Processor -- start 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee14e28700 1 -- start start 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee14e28700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee1010c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:35.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee14e28700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee100f6980 0x7fee1010e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee14e28700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10060810 con 0x7fee100f56e0 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee14e28700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10060950 con 0x7fee100f6980 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee1010c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee1010c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46402/0 (socket says 172.21.15.46:46402) 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0e59c700 1 -- 172.21.15.46:0/2781952900 learned_addr learned my addr 172.21.15.46:0/2781952900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0dd9b700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee100f6980 0x7fee1010e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0e59c700 1 -- 172.21.15.46:0/2781952900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee100f6980 msgr2=0x7fee1010e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0e59c700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee100f6980 0x7fee1010e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0e59c700 1 -- 172.21.15.46:0/2781952900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc005040 con 0x7fee100f56e0 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee0e59c700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee1010c220 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fedfc004e30 tx=0x7fedfc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee06ffd700 1 -- 172.21.15.46:0/2781952900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc018070 con 0x7fee100f56e0 2024-01-21T08:22:35.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1010ec10 con 0x7fee100f56e0 2024-01-21T08:22:35.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee06ffd700 1 -- 172.21.15.46:0/2781952900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedfc013410 con 0x7fee100f56e0 2024-01-21T08:22:35.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.017+0000 7fee06ffd700 1 -- 172.21.15.46:0/2781952900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc01de20 con 0x7fee100f56e0 2024-01-21T08:22:35.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.018+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1010efe0 con 0x7fee100f56e0 2024-01-21T08:22:35.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.018+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10045ad0 con 0x7fee100f56e0 2024-01-21T08:22:35.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.019+0000 7fee06ffd700 1 -- 172.21.15.46:0/2781952900 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fedfc01d420 con 0x7fee100f56e0 2024-01-21T08:22:35.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.019+0000 7fee06ffd700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fedf405c2a0 0x7fedf405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:35.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.019+0000 7fee06ffd700 1 -- 172.21.15.46:0/2781952900 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fedfc0777b0 con 0x7fee100f56e0 2024-01-21T08:22:35.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.019+0000 7fee0dd9b700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fedf405c2a0 0x7fedf405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:35.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.020+0000 7fee0dd9b700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fedf405c2a0 0x7fedf405e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fee00004c40 tx=0x7fee0000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:35.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.023+0000 7fee06ffd700 1 -- 172.21.15.46:0/2781952900 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedfc04a7b0 con 0x7fee100f56e0 2024-01-21T08:22:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.241+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fee10027a00 con 0x7fee100f56e0 2024-01-21T08:22:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.241+0000 7fee06ffd700 1 -- 172.21.15.46:0/2781952900 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fedfc04ddd0 con 0x7fee100f56e0 2024-01-21T08:22:35.243 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fedf405c2a0 msgr2=0x7fedf405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fedf405c2a0 0x7fedf405e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fee00004c40 tx=0x7fee0000b040).stop 2024-01-21T08:22:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 msgr2=0x7fee1010c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee1010c220 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fedfc004e30 tx=0x7fedfc009b30).stop 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 shutdown_connections 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fedf405c2a0 0x7fedf405e760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee100f56e0 0x7fee1010c220 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 --2- 172.21.15.46:0/2781952900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee100f6980 0x7fee1010e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 >> 172.21.15.46:0/2781952900 conn(0x7fee100f0e20 msgr2=0x7fee100fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 shutdown_connections 2024-01-21T08:22:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:35.243+0000 7fee14e28700 1 -- 172.21.15.46:0/2781952900 wait complete. 2024-01-21T08:22:35.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:34.144942+0000 mgr.smithi046.lmxnwc (mgr.14164) 582 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:35.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:34.144942+0000 mgr.smithi046.lmxnwc (mgr.14164) 582 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:35.632 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:36.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:36 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:35.242002+0000 mon.smithi046 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.46:0/2781952900' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:36.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:36 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:35.242002+0000 mon.smithi046 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.46:0/2781952900' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:37.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:36.145423+0000 mgr.smithi046.lmxnwc (mgr.14164) 583 : cluster [DBG] pgmap v544: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:37.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:36.145423+0000 mgr.smithi046.lmxnwc (mgr.14164) 583 : cluster [DBG] pgmap v544: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:38.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:38.879 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:39.327 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:38.145943+0000 mgr.smithi046.lmxnwc (mgr.14164) 584 : cluster [DBG] pgmap v545: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:39.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:38.145943+0000 mgr.smithi046.lmxnwc (mgr.14164) 584 : cluster [DBG] pgmap v545: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- 172.21.15.46:0/2477421572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f4a70 msgr2=0x7f54bc0f4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 --2- 172.21.15.46:0/2477421572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f4a70 0x7f54bc0f4e90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f54ac004660 tx=0x7f54ac009b30).stop 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- 172.21.15.46:0/2477421572 shutdown_connections 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 --2- 172.21.15.46:0/2477421572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54bc0f5410 0x7f54bc0f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 --2- 172.21.15.46:0/2477421572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f4a70 0x7f54bc0f4e90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- 172.21.15.46:0/2477421572 >> 172.21.15.46:0/2477421572 conn(0x7f54bc0f0590 msgr2=0x7f54bc0f29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- 172.21.15.46:0/2477421572 shutdown_connections 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- 172.21.15.46:0/2477421572 wait complete. 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 Processor -- start 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- start start 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54bc0f4a70 0x7f54bc107140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f5410 0x7f54bc107630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc107be0 con 0x7f54bc0f5410 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.856+0000 7f54c37d7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc107d20 con 0x7f54bc0f4a70 2024-01-21T08:22:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c0d72700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f5410 0x7f54bc107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c0d72700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f5410 0x7f54bc107630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46418/0 (socket says 172.21.15.46:46418) 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c0d72700 1 -- 172.21.15.46:0/3367109378 learned_addr learned my addr 172.21.15.46:0/3367109378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c1573700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54bc0f4a70 0x7f54bc107140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c0d72700 1 -- 172.21.15.46:0/3367109378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54bc0f4a70 msgr2=0x7f54bc107140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c0d72700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54bc0f4a70 0x7f54bc107140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c0d72700 1 -- 172.21.15.46:0/3367109378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac005040 con 0x7f54bc0f5410 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c0d72700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f5410 0x7f54bc107630 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f54b8002700 tx=0x7f54b8009da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54b1ffb700 1 -- 172.21.15.46:0/3367109378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54b80079c0 con 0x7f54bc0f5410 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54b1ffb700 1 -- 172.21.15.46:0/3367109378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54b800f040 con 0x7f54bc0f5410 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc1086c0 con 0x7f54bc0f5410 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54b1ffb700 1 -- 172.21.15.46:0/3367109378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54b80147e0 con 0x7f54bc0f5410 2024-01-21T08:22:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.857+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc108c80 con 0x7f54bc0f5410 2024-01-21T08:22:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.858+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc045ad0 con 0x7f54bc0f5410 2024-01-21T08:22:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.858+0000 7f54b1ffb700 1 -- 172.21.15.46:0/3367109378 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f54b801b060 con 0x7f54bc0f5410 2024-01-21T08:22:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.859+0000 7f54b1ffb700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f54a805c2a0 0x7f54a805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.859+0000 7f54b1ffb700 1 -- 172.21.15.46:0/3367109378 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f54b8075940 con 0x7f54bc0f5410 2024-01-21T08:22:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.859+0000 7f54c1573700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f54a805c2a0 0x7f54a805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.859+0000 7f54c1573700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f54a805c2a0 0x7f54a805e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f54ac009f50 tx=0x7f54ac009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:39.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:39.863+0000 7f54b1ffb700 1 -- 172.21.15.46:0/3367109378 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54b80489c0 con 0x7f54bc0f5410 2024-01-21T08:22:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.081+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f54bc027a00 con 0x7f54bc0f5410 2024-01-21T08:22:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.081+0000 7f54b1ffb700 1 -- 172.21.15.46:0/3367109378 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f54b804bfe0 con 0x7f54bc0f5410 2024-01-21T08:22:40.085 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:40.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f54a805c2a0 msgr2=0x7f54a805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f54a805c2a0 0x7f54a805e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f54ac009f50 tx=0x7f54ac009b30).stop 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f5410 msgr2=0x7f54bc107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f5410 0x7f54bc107630 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f54b8002700 tx=0x7f54b8009da0).stop 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 shutdown_connections 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f54a805c2a0 0x7f54a805e760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54bc0f4a70 0x7f54bc107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 --2- 172.21.15.46:0/3367109378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54bc0f5410 0x7f54bc107630 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 >> 172.21.15.46:0/3367109378 conn(0x7f54bc0f0590 msgr2=0x7f54bc0fb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 shutdown_connections 2024-01-21T08:22:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:40.083+0000 7f54c37d7700 1 -- 172.21.15.46:0/3367109378 wait complete. 2024-01-21T08:22:40.448 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:40.083316+0000 mon.smithi046 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.46:0/3367109378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:40.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:40.146233+0000 mgr.smithi046.lmxnwc (mgr.14164) 585 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:40.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:40.083316+0000 mon.smithi046 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.46:0/3367109378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:40.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:40.146233+0000 mgr.smithi046.lmxnwc (mgr.14164) 585 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:43.449 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:43.483 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:42.146811+0000 mgr.smithi046.lmxnwc (mgr.14164) 586 : cluster [DBG] pgmap v547: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:43.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:42.146811+0000 mgr.smithi046.lmxnwc (mgr.14164) 586 : cluster [DBG] pgmap v547: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 -- 172.21.15.46:0/3391576240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f7a80 msgr2=0x7fbd240f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/3391576240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f7a80 0x7fbd240f7ee0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fbd0c004660 tx=0x7fbd0c009b30).stop 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 -- 172.21.15.46:0/3391576240 shutdown_connections 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/3391576240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f7a80 0x7fbd240f7ee0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/3391576240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd240f67e0 0x7fbd240f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 -- 172.21.15.46:0/3391576240 >> 172.21.15.46:0/3391576240 conn(0x7fbd240f1f40 msgr2=0x7fbd240f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 -- 172.21.15.46:0/3391576240 shutdown_connections 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.912+0000 7fbd2a45c700 1 -- 172.21.15.46:0/3391576240 wait complete. 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.913+0000 7fbd2a45c700 1 Processor -- start 2024-01-21T08:22:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.913+0000 7fbd2a45c700 1 -- start start 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.913+0000 7fbd2a45c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f67e0 0x7fbd241070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.913+0000 7fbd2a45c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd240f7a80 0x7fbd241075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.913+0000 7fbd2a45c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24107b60 con 0x7fbd240f67e0 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.913+0000 7fbd2a45c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24107ca0 con 0x7fbd240f7a80 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd23fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f67e0 0x7fbd241070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd23fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f67e0 0x7fbd241070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35424/0 (socket says 172.21.15.46:35424) 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd23fff700 1 -- 172.21.15.46:0/515695676 learned_addr learned my addr 172.21.15.46:0/515695676 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd237fe700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd240f7a80 0x7fbd241075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd23fff700 1 -- 172.21.15.46:0/515695676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd240f7a80 msgr2=0x7fbd241075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd23fff700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd240f7a80 0x7fbd241075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd23fff700 1 -- 172.21.15.46:0/515695676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c005040 con 0x7fbd240f67e0 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd237fe700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd240f7a80 0x7fbd241075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-21T08:22:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd23fff700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f67e0 0x7fbd241070c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fbd1400c530 tx=0x7fbd14007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd20ff9700 1 -- 172.21.15.46:0/515695676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1400f650 con 0x7fbd240f67e0 2024-01-21T08:22:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd20ff9700 1 -- 172.21.15.46:0/515695676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd1400fc60 con 0x7fbd240f67e0 2024-01-21T08:22:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd20ff9700 1 -- 172.21.15.46:0/515695676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd14012d20 con 0x7fbd240f67e0 2024-01-21T08:22:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd24108640 con 0x7fbd240f67e0 2024-01-21T08:22:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.914+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24108c00 con 0x7fbd240f67e0 2024-01-21T08:22:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.915+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24045ad0 con 0x7fbd240f67e0 2024-01-21T08:22:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.915+0000 7fbd20ff9700 1 -- 172.21.15.46:0/515695676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbd14019030 con 0x7fbd240f67e0 2024-01-21T08:22:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.916+0000 7fbd20ff9700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1005c2a0 0x7fbd1005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.916+0000 7fbd20ff9700 1 -- 172.21.15.46:0/515695676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbd140757a0 con 0x7fbd240f67e0 2024-01-21T08:22:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.916+0000 7fbd237fe700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1005c2a0 0x7fbd1005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.916+0000 7fbd237fe700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1005c2a0 0x7fbd1005e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbd0c013580 tx=0x7fbd0c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:44.920+0000 7fbd20ff9700 1 -- 172.21.15.46:0/515695676 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd140462b0 con 0x7fbd240f67e0 2024-01-21T08:22:45.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.138+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd240ff0f0 con 0x7fbd240f67e0 2024-01-21T08:22:45.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.139+0000 7fbd20ff9700 1 -- 172.21.15.46:0/515695676 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fbd14017080 con 0x7fbd240f67e0 2024-01-21T08:22:45.141 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1005c2a0 msgr2=0x7fbd1005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1005c2a0 0x7fbd1005e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbd0c013580 tx=0x7fbd0c009b30).stop 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f67e0 msgr2=0x7fbd241070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f67e0 0x7fbd241070c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fbd1400c530 tx=0x7fbd14007e10).stop 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 shutdown_connections 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd1005c2a0 0x7fbd1005e760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd240f67e0 0x7fbd241070c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 --2- 172.21.15.46:0/515695676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd240f7a80 0x7fbd241075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 >> 172.21.15.46:0/515695676 conn(0x7fbd240f1f40 msgr2=0x7fbd240fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 shutdown_connections 2024-01-21T08:22:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:45.140+0000 7fbd2a45c700 1 -- 172.21.15.46:0/515695676 wait complete. 2024-01-21T08:22:45.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:44.147344+0000 mgr.smithi046.lmxnwc (mgr.14164) 587 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:45.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:45.140520+0000 mon.smithi046 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.46:0/515695676' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:45.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:44.147344+0000 mgr.smithi046.lmxnwc (mgr.14164) 587 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:45.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:45.140520+0000 mon.smithi046 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.46:0/515695676' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:45.556 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:47.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:46.147915+0000 mgr.smithi046.lmxnwc (mgr.14164) 588 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:47.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:46.147915+0000 mgr.smithi046.lmxnwc (mgr.14164) 588 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:48.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:48.803 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:49.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:48.148499+0000 mgr.smithi046.lmxnwc (mgr.14164) 589 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:49.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:48.148499+0000 mgr.smithi046.lmxnwc (mgr.14164) 589 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 -- 172.21.15.46:0/1312100433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac0f9920 msgr2=0x7fadac0fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 --2- 172.21.15.46:0/1312100433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac0f9920 0x7fadac0fbd70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fada0004660 tx=0x7fada0009b30).stop 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 -- 172.21.15.46:0/1312100433 shutdown_connections 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 --2- 172.21.15.46:0/1312100433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac0f9920 0x7fadac0fbd70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 --2- 172.21.15.46:0/1312100433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac060870 0x7fadac0f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 -- 172.21.15.46:0/1312100433 >> 172.21.15.46:0/1312100433 conn(0x7fadac0f0e20 msgr2=0x7fadac0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 -- 172.21.15.46:0/1312100433 shutdown_connections 2024-01-21T08:22:49.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.832+0000 7fadb1f33700 1 -- 172.21.15.46:0/1312100433 wait complete. 2024-01-21T08:22:49.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadb1f33700 1 Processor -- start 2024-01-21T08:22:49.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadb1f33700 1 -- start start 2024-01-21T08:22:49.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadb1f33700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac060870 0x7fadac1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadb1f33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0f9920 0x7fadac107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadb1f33700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac107b40 con 0x7fadac060870 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadb1f33700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac107c80 con 0x7fadac0f9920 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac060870 0x7fadac1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac060870 0x7fadac1070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35444/0 (socket says 172.21.15.46:35444) 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadab7fe700 1 -- 172.21.15.46:0/2929068484 learned_addr learned my addr 172.21.15.46:0/2929068484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadaaffd700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0f9920 0x7fadac107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.833+0000 7fadab7fe700 1 -- 172.21.15.46:0/2929068484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0f9920 msgr2=0x7fadac107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fadab7fe700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0f9920 0x7fadac107590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fadab7fe700 1 -- 172.21.15.46:0/2929068484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada0005040 con 0x7fadac060870 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fadab7fe700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac060870 0x7fadac1070a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fad9c0026d0 tx=0x7fad9c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fad93fff700 1 -- 172.21.15.46:0/2929068484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad9c009320 con 0x7fadac060870 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fad93fff700 1 -- 172.21.15.46:0/2929068484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad9c00f040 con 0x7fadac060870 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac108620 con 0x7fadac060870 2024-01-21T08:22:49.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fad93fff700 1 -- 172.21.15.46:0/2929068484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad9c0147e0 con 0x7fadac060870 2024-01-21T08:22:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.834+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac108be0 con 0x7fadac060870 2024-01-21T08:22:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.835+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac045ad0 con 0x7fadac060870 2024-01-21T08:22:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.835+0000 7fad93fff700 1 -- 172.21.15.46:0/2929068484 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fad9c009480 con 0x7fadac060870 2024-01-21T08:22:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.836+0000 7fad93fff700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad9405c2a0 0x7fad9405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.836+0000 7fad93fff700 1 -- 172.21.15.46:0/2929068484 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fad9c076270 con 0x7fadac060870 2024-01-21T08:22:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.836+0000 7fadaaffd700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad9405c2a0 0x7fad9405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.836+0000 7fadaaffd700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad9405c2a0 0x7fad9405e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fada0009d60 tx=0x7fada0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:49.840+0000 7fad93fff700 1 -- 172.21.15.46:0/2929068484 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad9c049270 con 0x7fadac060870 2024-01-21T08:22:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.058+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fadac0ff0d0 con 0x7fadac060870 2024-01-21T08:22:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.059+0000 7fad93fff700 1 -- 172.21.15.46:0/2929068484 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fad9c04c890 con 0x7fadac060870 2024-01-21T08:22:50.061 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:50.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad9405c2a0 msgr2=0x7fad9405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:50.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad9405c2a0 0x7fad9405e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fada0009d60 tx=0x7fada0009b30).stop 2024-01-21T08:22:50.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac060870 msgr2=0x7fadac1070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac060870 0x7fadac1070a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fad9c0026d0 tx=0x7fad9c009de0).stop 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 shutdown_connections 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad9405c2a0 0x7fad9405e760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadac060870 0x7fadac1070a0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 --2- 172.21.15.46:0/2929068484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0f9920 0x7fadac107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 >> 172.21.15.46:0/2929068484 conn(0x7fadac0f0e20 msgr2=0x7fadac0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 shutdown_connections 2024-01-21T08:22:50.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:50.061+0000 7fadb1f33700 1 -- 172.21.15.46:0/2929068484 wait complete. 2024-01-21T08:22:50.428 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:50.060736+0000 mon.smithi046 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.46:0/2929068484' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:50.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:50.148910+0000 mgr.smithi046.lmxnwc (mgr.14164) 590 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:50.060736+0000 mon.smithi046 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.46:0/2929068484' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:50.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:50.148910+0000 mgr.smithi046.lmxnwc (mgr.14164) 590 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:53.429 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:53.464 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:52.149269+0000 mgr.smithi046.lmxnwc (mgr.14164) 591 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:53.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:52.149269+0000 mgr.smithi046.lmxnwc (mgr.14164) 591 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 -- 172.21.15.46:0/1642658184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 msgr2=0x7f32ac0f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 --2- 172.21.15.46:0/1642658184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac0f7ed0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f3298004660 tx=0x7f3298009b30).stop 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 -- 172.21.15.46:0/1642658184 shutdown_connections 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 --2- 172.21.15.46:0/1642658184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac0f7ed0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 --2- 172.21.15.46:0/1642658184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32ac0f67d0 0x7f32ac0f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 -- 172.21.15.46:0/1642658184 >> 172.21.15.46:0/1642658184 conn(0x7f32ac0f1f50 msgr2=0x7f32ac0f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 -- 172.21.15.46:0/1642658184 shutdown_connections 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.679+0000 7f32b3076700 1 -- 172.21.15.46:0/1642658184 wait complete. 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.680+0000 7f32b3076700 1 Processor -- start 2024-01-21T08:22:54.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.680+0000 7f32b3076700 1 -- start start 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.680+0000 7f32b3076700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32ac0f67d0 0x7f32ac1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.680+0000 7f32b3076700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac1075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.680+0000 7f32b3076700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac107bf0 con 0x7f32ac0f7a70 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.680+0000 7f32b3076700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac1084a0 con 0x7f32ac0f67d0 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.680+0000 7f32abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac1075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac1075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41062/0 (socket says 172.21.15.46:41062) 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32abfff700 1 -- 172.21.15.46:0/4008101133 learned_addr learned my addr 172.21.15.46:0/4008101133 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32b0e12700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32ac0f67d0 0x7f32ac1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32abfff700 1 -- 172.21.15.46:0/4008101133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32ac0f67d0 msgr2=0x7f32ac1070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32abfff700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32ac0f67d0 0x7f32ac1070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32abfff700 1 -- 172.21.15.46:0/4008101133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298005040 con 0x7f32ac0f7a70 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32abfff700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac1075b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f3298009830 tx=0x7f3298009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32a97fa700 1 -- 172.21.15.46:0/4008101133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3298018070 con 0x7f32ac0f7a70 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac108720 con 0x7f32ac0f7a70 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32a97fa700 1 -- 172.21.15.46:0/4008101133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3298013410 con 0x7f32ac0f7a70 2024-01-21T08:22:54.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32a97fa700 1 -- 172.21.15.46:0/4008101133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329801c620 con 0x7f32ac0f7a70 2024-01-21T08:22:54.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.681+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac108c80 con 0x7f32ac0f7a70 2024-01-21T08:22:54.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.682+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac059100 con 0x7f32ac0f7a70 2024-01-21T08:22:54.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.683+0000 7f32a97fa700 1 -- 172.21.15.46:0/4008101133 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f329801c8e0 con 0x7f32ac0f7a70 2024-01-21T08:22:54.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.683+0000 7f32a97fa700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f329c05c2a0 0x7f329c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:54.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.683+0000 7f32a97fa700 1 -- 172.21.15.46:0/4008101133 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3298077a80 con 0x7f32ac0f7a70 2024-01-21T08:22:54.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.683+0000 7f32b0e12700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f329c05c2a0 0x7f329c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:54.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.683+0000 7f32b0e12700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f329c05c2a0 0x7f329c05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f32ac0f78d0 tx=0x7f32a0008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:54.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.687+0000 7f32a97fa700 1 -- 172.21.15.46:0/4008101133 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f329804aa20 con 0x7f32ac0f7a70 2024-01-21T08:22:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.906+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f32ac0ff000 con 0x7f32ac0f7a70 2024-01-21T08:22:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.906+0000 7f32a97fa700 1 -- 172.21.15.46:0/4008101133 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f329804e040 con 0x7f32ac0f7a70 2024-01-21T08:22:54.909 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.907+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f329c05c2a0 msgr2=0x7f329c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.907+0000 7f32b3076700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f329c05c2a0 0x7f329c05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f32ac0f78d0 tx=0x7f32a0008040).stop 2024-01-21T08:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.907+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 msgr2=0x7f32ac1075b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.907+0000 7f32b3076700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac1075b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f3298009830 tx=0x7f3298009b30).stop 2024-01-21T08:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.908+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 shutdown_connections 2024-01-21T08:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.908+0000 7f32b3076700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f329c05c2a0 0x7f329c05e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.908+0000 7f32b3076700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32ac0f67d0 0x7f32ac1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.908+0000 7f32b3076700 1 --2- 172.21.15.46:0/4008101133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32ac0f7a70 0x7f32ac1075b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.908+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 >> 172.21.15.46:0/4008101133 conn(0x7f32ac0f1f50 msgr2=0x7f32ac0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.908+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 shutdown_connections 2024-01-21T08:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:54.908+0000 7f32b3076700 1 -- 172.21.15.46:0/4008101133 wait complete. 2024-01-21T08:22:55.287 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:22:55.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:54.149571+0000 mgr.smithi046.lmxnwc (mgr.14164) 592 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:55.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:54.907822+0000 mon.smithi046 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.46:0/4008101133' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:55.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:54.149571+0000 mgr.smithi046.lmxnwc (mgr.14164) 592 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:55.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:54.907822+0000 mon.smithi046 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.46:0/4008101133' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:22:57.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:56.150091+0000 mgr.smithi046.lmxnwc (mgr.14164) 593 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:57.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:56.150091+0000 mgr.smithi046.lmxnwc (mgr.14164) 593 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:58.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:22:58.534 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:22:59.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:22:59 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:22:58.150777+0000 mgr.smithi046.lmxnwc (mgr.14164) 594 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:59.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:22:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:22:58.150777+0000 mgr.smithi046.lmxnwc (mgr.14164) 594 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- 172.21.15.46:0/468200884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 msgr2=0x7f41bc0fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 --2- 172.21.15.46:0/468200884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc0fbbc0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f41ac004660 tx=0x7f41ac009b30).stop 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- 172.21.15.46:0/468200884 shutdown_connections 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 --2- 172.21.15.46:0/468200884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc0fbbc0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 --2- 172.21.15.46:0/468200884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41bc060870 0x7f41bc0f91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- 172.21.15.46:0/468200884 >> 172.21.15.46:0/468200884 conn(0x7f41bc0f0c70 msgr2=0x7f41bc0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- 172.21.15.46:0/468200884 shutdown_connections 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- 172.21.15.46:0/468200884 wait complete. 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 Processor -- start 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- start start 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41bc060870 0x7f41bc106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc1073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:59.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc107970 con 0x7f41bc0f9770 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.496+0000 7f41c24df700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc107ab0 con 0x7f41bc060870 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c0cdc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc1073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c0cdc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc1073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41074/0 (socket says 172.21.15.46:41074) 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c0cdc700 1 -- 172.21.15.46:0/61831996 learned_addr learned my addr 172.21.15.46:0/61831996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c14dd700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41bc060870 0x7f41bc106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c0cdc700 1 -- 172.21.15.46:0/61831996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41bc060870 msgr2=0x7f41bc106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c0cdc700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41bc060870 0x7f41bc106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c0cdc700 1 -- 172.21.15.46:0/61831996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac005040 con 0x7f41bc0f9770 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c0cdc700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc1073c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f41ac004e30 tx=0x7f41ac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41b1ffb700 1 -- 172.21.15.46:0/61831996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ac018070 con 0x7f41bc0f9770 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41b1ffb700 1 -- 172.21.15.46:0/61831996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41ac013410 con 0x7f41bc0f9770 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc1083f0 con 0x7f41bc0f9770 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41b1ffb700 1 -- 172.21.15.46:0/61831996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ac01de20 con 0x7f41bc0f9770 2024-01-21T08:22:59.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.497+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc108950 con 0x7f41bc0f9770 2024-01-21T08:22:59.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.498+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc045ad0 con 0x7f41bc0f9770 2024-01-21T08:22:59.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.499+0000 7f41b1ffb700 1 -- 172.21.15.46:0/61831996 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f41ac01d420 con 0x7f41bc0f9770 2024-01-21T08:22:59.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.499+0000 7f41b1ffb700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f41a805c2b0 0x7f41a805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:22:59.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.499+0000 7f41b1ffb700 1 -- 172.21.15.46:0/61831996 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f41ac077730 con 0x7f41bc0f9770 2024-01-21T08:22:59.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.499+0000 7f41c14dd700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f41a805c2b0 0x7f41a805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:22:59.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.500+0000 7f41c14dd700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f41a805c2b0 0x7f41a805e770 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f41b8004c40 tx=0x7f41b800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:22:59.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.503+0000 7f41b1ffb700 1 -- 172.21.15.46:0/61831996 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41ac04a7b0 con 0x7f41bc0f9770 2024-01-21T08:22:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.724+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f41bc027a00 con 0x7f41bc0f9770 2024-01-21T08:22:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.725+0000 7f41b1ffb700 1 -- 172.21.15.46:0/61831996 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f41bc027a00 con 0x7f41bc0f9770 2024-01-21T08:22:59.727 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f41a805c2b0 msgr2=0x7f41a805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f41a805c2b0 0x7f41a805e770 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f41b8004c40 tx=0x7f41b800b040).stop 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 msgr2=0x7f41bc1073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc1073c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f41ac004e30 tx=0x7f41ac009b30).stop 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 shutdown_connections 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f41a805c2b0 0x7f41a805e770 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41bc060870 0x7f41bc106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 --2- 172.21.15.46:0/61831996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41bc0f9770 0x7f41bc1073c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:22:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 >> 172.21.15.46:0/61831996 conn(0x7f41bc0f0c70 msgr2=0x7f41bc0f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:22:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 shutdown_connections 2024-01-21T08:22:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:22:59.726+0000 7f41c24df700 1 -- 172.21.15.46:0/61831996 wait complete. 2024-01-21T08:23:00.103 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:22:59.726513+0000 mon.smithi046 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.46:0/61831996' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:00.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:00.151146+0000 mgr.smithi046.lmxnwc (mgr.14164) 595 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:22:59.726513+0000 mon.smithi046 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.46:0/61831996' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:00.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:00 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:00.151146+0000 mgr.smithi046.lmxnwc (mgr.14164) 595 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:03.105 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:03.465 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:03 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:02.151661+0000 mgr.smithi046.lmxnwc (mgr.14164) 596 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:03.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:03 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:02.151661+0000 mgr.smithi046.lmxnwc (mgr.14164) 596 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- 172.21.15.46:0/1719046412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 msgr2=0x7f36c00f7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 --2- 172.21.15.46:0/1719046412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c00f7d20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f36b0004660 tx=0x7f36b0009b30).stop 2024-01-21T08:23:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- 172.21.15.46:0/1719046412 shutdown_connections 2024-01-21T08:23:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 --2- 172.21.15.46:0/1719046412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c00f7d20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 --2- 172.21.15.46:0/1719046412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36c00f6620 0x7f36c00f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- 172.21.15.46:0/1719046412 >> 172.21.15.46:0/1719046412 conn(0x7f36c00f1da0 msgr2=0x7f36c00f41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- 172.21.15.46:0/1719046412 shutdown_connections 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- 172.21.15.46:0/1719046412 wait complete. 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 Processor -- start 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- start start 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36c00f6620 0x7f36c0106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c01073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c0107980 con 0x7f36c00f78c0 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c619a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c0107ac0 con 0x7f36c00f6620 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.298+0000 7f36c4997700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c01073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c4997700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c01073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56508/0 (socket says 172.21.15.46:56508) 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c4997700 1 -- 172.21.15.46:0/1590554962 learned_addr learned my addr 172.21.15.46:0/1590554962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c5198700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36c00f6620 0x7f36c0106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c4997700 1 -- 172.21.15.46:0/1590554962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36c00f6620 msgr2=0x7f36c0106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c4997700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36c00f6620 0x7f36c0106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c4997700 1 -- 172.21.15.46:0/1590554962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0005040 con 0x7f36c00f78c0 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c4997700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c01073d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f36b0009830 tx=0x7f36b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36b5ffb700 1 -- 172.21.15.46:0/1590554962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b0018070 con 0x7f36c00f78c0 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c0108400 con 0x7f36c00f78c0 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36b5ffb700 1 -- 172.21.15.46:0/1590554962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36b0013410 con 0x7f36c00f78c0 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36b5ffb700 1 -- 172.21.15.46:0/1590554962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b001c400 con 0x7f36c00f78c0 2024-01-21T08:23:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.299+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c0108960 con 0x7f36c00f78c0 2024-01-21T08:23:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.300+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c0045ad0 con 0x7f36c00f78c0 2024-01-21T08:23:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.301+0000 7f36b5ffb700 1 -- 172.21.15.46:0/1590554962 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f36b001d420 con 0x7f36c00f78c0 2024-01-21T08:23:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.301+0000 7f36b5ffb700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f36ac05c2b0 0x7f36ac05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.301+0000 7f36b5ffb700 1 -- 172.21.15.46:0/1590554962 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f36b0077a30 con 0x7f36c00f78c0 2024-01-21T08:23:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.301+0000 7f36c5198700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f36ac05c2b0 0x7f36ac05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.301+0000 7f36c5198700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f36ac05c2b0 0x7f36ac05e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f36c00f7720 tx=0x7f36bc009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.305+0000 7f36b5ffb700 1 -- 172.21.15.46:0/1590554962 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36b004aa30 con 0x7f36c00f78c0 2024-01-21T08:23:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.524+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f36c0027a00 con 0x7f36c00f78c0 2024-01-21T08:23:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.524+0000 7f36b5ffb700 1 -- 172.21.15.46:0/1590554962 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f36b004e050 con 0x7f36c00f78c0 2024-01-21T08:23:04.527 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f36ac05c2b0 msgr2=0x7f36ac05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f36ac05c2b0 0x7f36ac05e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f36c00f7720 tx=0x7f36bc009040).stop 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 msgr2=0x7f36c01073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c01073d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f36b0009830 tx=0x7f36b0009b30).stop 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 shutdown_connections 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f36ac05c2b0 0x7f36ac05e770 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36c00f6620 0x7f36c0106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 --2- 172.21.15.46:0/1590554962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c00f78c0 0x7f36c01073d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 >> 172.21.15.46:0/1590554962 conn(0x7f36c00f1da0 msgr2=0x7f36c00fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 shutdown_connections 2024-01-21T08:23:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:04.526+0000 7f36c619a700 1 -- 172.21.15.46:0/1590554962 wait complete. 2024-01-21T08:23:04.911 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:05.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:04.152076+0000 mgr.smithi046.lmxnwc (mgr.14164) 597 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:05.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:05 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:04.526261+0000 mon.smithi046 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.46:0/1590554962' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:05.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:04.152076+0000 mgr.smithi046.lmxnwc (mgr.14164) 597 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:05.492 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:04.526261+0000 mon.smithi046 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.46:0/1590554962' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:07.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:07 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:06.152433+0000 mgr.smithi046.lmxnwc (mgr.14164) 598 : cluster [DBG] pgmap v559: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:07.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:07 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:06.152433+0000 mgr.smithi046.lmxnwc (mgr.14164) 598 : cluster [DBG] pgmap v559: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:07.913 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.159+0000 7fbd4f917700 1 -- 172.21.15.46:0/815844637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f56c0 msgr2=0x7fbd480f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.159+0000 7fbd4f917700 1 --2- 172.21.15.46:0/815844637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f56c0 0x7fbd480f5ae0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fbd44004660 tx=0x7fbd44009b30).stop 2024-01-21T08:23:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.159+0000 7fbd4f917700 1 -- 172.21.15.46:0/815844637 shutdown_connections 2024-01-21T08:23:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.159+0000 7fbd4f917700 1 --2- 172.21.15.46:0/815844637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd480f6960 0x7fbd480f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.159+0000 7fbd4f917700 1 --2- 172.21.15.46:0/815844637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f56c0 0x7fbd480f5ae0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.159+0000 7fbd4f917700 1 -- 172.21.15.46:0/815844637 >> 172.21.15.46:0/815844637 conn(0x7fbd480f0e20 msgr2=0x7fbd480f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.159+0000 7fbd4f917700 1 -- 172.21.15.46:0/815844637 shutdown_connections 2024-01-21T08:23:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.160+0000 7fbd4f917700 1 -- 172.21.15.46:0/815844637 wait complete. 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.160+0000 7fbd4f917700 1 Processor -- start 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.160+0000 7fbd4f917700 1 -- start start 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.160+0000 7fbd4f917700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd480f56c0 0x7fbd48109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.160+0000 7fbd4f917700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f6960 0x7fbd4810c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4f917700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48060920 con 0x7fbd480f6960 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4f917700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48060a60 con 0x7fbd480f56c0 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4ceb2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f6960 0x7fbd4810c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4ceb2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f6960 0x7fbd4810c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56522/0 (socket says 172.21.15.46:56522) 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4ceb2700 1 -- 172.21.15.46:0/50354883 learned_addr learned my addr 172.21.15.46:0/50354883 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4d6b3700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd480f56c0 0x7fbd48109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4ceb2700 1 -- 172.21.15.46:0/50354883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd480f56c0 msgr2=0x7fbd48109df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4ceb2700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd480f56c0 0x7fbd48109df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4ceb2700 1 -- 172.21.15.46:0/50354883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd44005040 con 0x7fbd480f6960 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4ceb2700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f6960 0x7fbd4810c2f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7fbd3800c530 tx=0x7fbd38007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd3dffb700 1 -- 172.21.15.46:0/50354883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd3800f650 con 0x7fbd480f6960 2024-01-21T08:23:09.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd3dffb700 1 -- 172.21.15.46:0/50354883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd3800fc60 con 0x7fbd480f6960 2024-01-21T08:23:09.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4810c8a0 con 0x7fbd480f6960 2024-01-21T08:23:09.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.161+0000 7fbd3dffb700 1 -- 172.21.15.46:0/50354883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd38012c30 con 0x7fbd480f6960 2024-01-21T08:23:09.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.162+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4810ce30 con 0x7fbd480f6960 2024-01-21T08:23:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.162+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd48059100 con 0x7fbd480f6960 2024-01-21T08:23:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.163+0000 7fbd3dffb700 1 -- 172.21.15.46:0/50354883 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbd38010070 con 0x7fbd480f6960 2024-01-21T08:23:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.163+0000 7fbd3dffb700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd3405c2f0 0x7fbd3405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.163+0000 7fbd3dffb700 1 -- 172.21.15.46:0/50354883 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbd38075810 con 0x7fbd480f6960 2024-01-21T08:23:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.163+0000 7fbd4d6b3700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd3405c2f0 0x7fbd3405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.164+0000 7fbd4d6b3700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd3405c2f0 0x7fbd3405e7b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fbd480605a0 tx=0x7fbd44009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.167+0000 7fbd3dffb700 1 -- 172.21.15.46:0/50354883 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd380452b0 con 0x7fbd480f6960 2024-01-21T08:23:09.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:09 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:08.152775+0000 mgr.smithi046.lmxnwc (mgr.14164) 599 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:09.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.388+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd48045ad0 con 0x7fbd480f6960 2024-01-21T08:23:09.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.389+0000 7fbd3dffb700 1 -- 172.21.15.46:0/50354883 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fbd380487d0 con 0x7fbd480f6960 2024-01-21T08:23:09.392 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd3405c2f0 msgr2=0x7fbd3405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd3405c2f0 0x7fbd3405e7b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fbd480605a0 tx=0x7fbd44009b30).stop 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f6960 msgr2=0x7fbd4810c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f6960 0x7fbd4810c2f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7fbd3800c530 tx=0x7fbd38007e20).stop 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 shutdown_connections 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fbd3405c2f0 0x7fbd3405e7b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd480f56c0 0x7fbd48109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 --2- 172.21.15.46:0/50354883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd480f6960 0x7fbd4810c2f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.390+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 >> 172.21.15.46:0/50354883 conn(0x7fbd480f0e20 msgr2=0x7fbd480fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:09.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.391+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 shutdown_connections 2024-01-21T08:23:09.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:09.391+0000 7fbd4f917700 1 -- 172.21.15.46:0/50354883 wait complete. 2024-01-21T08:23:09.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:09 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:08.152775+0000 mgr.smithi046.lmxnwc (mgr.14164) 599 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:09.769 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:10.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:10 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:09.390623+0000 mon.smithi046 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.46:0/50354883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:10.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:10 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:09.390623+0000 mon.smithi046 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.46:0/50354883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:11.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:11 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:10.153212+0000 mgr.smithi046.lmxnwc (mgr.14164) 600 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:11.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:11 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:10.153212+0000 mgr.smithi046.lmxnwc (mgr.14164) 600 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:12.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:13.017 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:13.456 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:13 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:12.153723+0000 mgr.smithi046.lmxnwc (mgr.14164) 601 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:13.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:13 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:12.153723+0000 mgr.smithi046.lmxnwc (mgr.14164) 601 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 -- 172.21.15.46:0/3264073059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f56c0 msgr2=0x7f78540f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 --2- 172.21.15.46:0/3264073059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f56c0 0x7f78540f5ae0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f7844004660 tx=0x7f7844009b30).stop 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 -- 172.21.15.46:0/3264073059 shutdown_connections 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 --2- 172.21.15.46:0/3264073059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78540f6960 0x7f78540f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 --2- 172.21.15.46:0/3264073059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f56c0 0x7f78540f5ae0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 -- 172.21.15.46:0/3264073059 >> 172.21.15.46:0/3264073059 conn(0x7f78540f0e20 msgr2=0x7f78540f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 -- 172.21.15.46:0/3264073059 shutdown_connections 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 -- 172.21.15.46:0/3264073059 wait complete. 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.969+0000 7f785a55b700 1 Processor -- start 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f785a55b700 1 -- start start 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f785a55b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78540f56c0 0x7f78540f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f785a55b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f6960 0x7f78540f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f785a55b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540f8200 con 0x7f78540f6960 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f785a55b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540f8340 con 0x7f78540f56c0 2024-01-21T08:23:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f78537fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f6960 0x7f78540f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f78537fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f6960 0x7f78540f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59456/0 (socket says 172.21.15.46:59456) 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f78537fe700 1 -- 172.21.15.46:0/744229435 learned_addr learned my addr 172.21.15.46:0/744229435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f7853fff700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78540f56c0 0x7f78540f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f78537fe700 1 -- 172.21.15.46:0/744229435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78540f56c0 msgr2=0x7f78540f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f78537fe700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78540f56c0 0x7f78540f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f78537fe700 1 -- 172.21.15.46:0/744229435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7844005040 con 0x7f78540f6960 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.970+0000 7f78537fe700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f6960 0x7f78540f7ca0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f78480026d0 tx=0x7f7848009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.971+0000 7f7850ff9700 1 -- 172.21.15.46:0/744229435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78480079c0 con 0x7f78540f6960 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.971+0000 7f7850ff9700 1 -- 172.21.15.46:0/744229435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f784800f040 con 0x7f78540f6960 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.971+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78540f8620 con 0x7f78540f6960 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.971+0000 7f7850ff9700 1 -- 172.21.15.46:0/744229435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78480148b0 con 0x7f78540f6960 2024-01-21T08:23:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.971+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785410f120 con 0x7f78540f6960 2024-01-21T08:23:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.972+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7854045ad0 con 0x7f78540f6960 2024-01-21T08:23:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.972+0000 7f7850ff9700 1 -- 172.21.15.46:0/744229435 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7848007b20 con 0x7f78540f6960 2024-01-21T08:23:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.972+0000 7f7850ff9700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f783c05c2a0 0x7f783c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.973+0000 7f7850ff9700 1 -- 172.21.15.46:0/744229435 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7848075b00 con 0x7f78540f6960 2024-01-21T08:23:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.973+0000 7f7853fff700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f783c05c2a0 0x7f783c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.973+0000 7f7853fff700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f783c05c2a0 0x7f783c05e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f78540f95b0 tx=0x7f7844009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:13.977+0000 7f7850ff9700 1 -- 172.21.15.46:0/744229435 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7848048b00 con 0x7f78540f6960 2024-01-21T08:23:14.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.194+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f78540fde50 con 0x7f78540f6960 2024-01-21T08:23:14.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.195+0000 7f7850ff9700 1 -- 172.21.15.46:0/744229435 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f784804c120 con 0x7f78540f6960 2024-01-21T08:23:14.197 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.196+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f783c05c2a0 msgr2=0x7f783c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.196+0000 7f785a55b700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f783c05c2a0 0x7f783c05e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f78540f95b0 tx=0x7f7844009b30).stop 2024-01-21T08:23:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.196+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f6960 msgr2=0x7f78540f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.196+0000 7f785a55b700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f6960 0x7f78540f7ca0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f78480026d0 tx=0x7f7848009de0).stop 2024-01-21T08:23:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.197+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 shutdown_connections 2024-01-21T08:23:14.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.197+0000 7f785a55b700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f783c05c2a0 0x7f783c05e760 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:14.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.197+0000 7f785a55b700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78540f56c0 0x7f78540f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:14.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.197+0000 7f785a55b700 1 --2- 172.21.15.46:0/744229435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78540f6960 0x7f78540f7ca0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:14.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.197+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 >> 172.21.15.46:0/744229435 conn(0x7f78540f0e20 msgr2=0x7f78540603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:14.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.197+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 shutdown_connections 2024-01-21T08:23:14.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:14.197+0000 7f785a55b700 1 -- 172.21.15.46:0/744229435 wait complete. 2024-01-21T08:23:14.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:14 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:14.195876+0000 mon.smithi046 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.46:0/744229435' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:14.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:14 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:14.195876+0000 mon.smithi046 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.46:0/744229435' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:14.577 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:15.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:15 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:14.154079+0000 mgr.smithi046.lmxnwc (mgr.14164) 602 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:15.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:15 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:14.154079+0000 mgr.smithi046.lmxnwc (mgr.14164) 602 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:17.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:17 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:16.154557+0000 mgr.smithi046.lmxnwc (mgr.14164) 603 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:17.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:17 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:16.154557+0000 mgr.smithi046.lmxnwc (mgr.14164) 603 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:17.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:17.822 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 -- 172.21.15.46:0/1390379403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 msgr2=0x7f45c00f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 --2- 172.21.15.46:0/1390379403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c00f5ae0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f45bc004660 tx=0x7f45bc009b30).stop 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 -- 172.21.15.46:0/1390379403 shutdown_connections 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 --2- 172.21.15.46:0/1390379403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45c00f6960 0x7f45c00f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 --2- 172.21.15.46:0/1390379403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c00f5ae0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 -- 172.21.15.46:0/1390379403 >> 172.21.15.46:0/1390379403 conn(0x7f45c00f0e20 msgr2=0x7f45c00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 -- 172.21.15.46:0/1390379403 shutdown_connections 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 -- 172.21.15.46:0/1390379403 wait complete. 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 Processor -- start 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.762+0000 7f45c794f700 1 -- start start 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c794f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c0106000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c794f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45c00f6960 0x7f45c01064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c794f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c0106aa0 con 0x7f45c00f56c0 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c794f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c0106be0 con 0x7f45c00f6960 2024-01-21T08:23:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c56eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c0106000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c56eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c0106000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59480/0 (socket says 172.21.15.46:59480) 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c56eb700 1 -- 172.21.15.46:0/1047790976 learned_addr learned my addr 172.21.15.46:0/1047790976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c56eb700 1 -- 172.21.15.46:0/1047790976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45c00f6960 msgr2=0x7f45c01064f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c4eea700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45c00f6960 0x7f45c01064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c56eb700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45c00f6960 0x7f45c01064f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c56eb700 1 -- 172.21.15.46:0/1047790976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc005040 con 0x7f45c00f56c0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c4eea700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45c00f6960 0x7f45c01064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c56eb700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c0106000 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f45bc009f50 tx=0x7f45bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45b1ffb700 1 -- 172.21.15.46:0/1047790976 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45bc018070 con 0x7f45c00f56c0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45b1ffb700 1 -- 172.21.15.46:0/1047790976 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45bc013410 con 0x7f45c00f56c0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45b1ffb700 1 -- 172.21.15.46:0/1047790976 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45bc01c620 con 0x7f45c00f56c0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.763+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c0107520 con 0x7f45c00f56c0 2024-01-21T08:23:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.764+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c0107a80 con 0x7f45c00f56c0 2024-01-21T08:23:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.765+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c0045ad0 con 0x7f45c00f56c0 2024-01-21T08:23:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.765+0000 7f45b1ffb700 1 -- 172.21.15.46:0/1047790976 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f45bc01d420 con 0x7f45c00f56c0 2024-01-21T08:23:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.765+0000 7f45b1ffb700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f45ac05c2a0 0x7f45ac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.765+0000 7f45b1ffb700 1 -- 172.21.15.46:0/1047790976 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f45bc0779f0 con 0x7f45c00f56c0 2024-01-21T08:23:18.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.765+0000 7f45c4eea700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f45ac05c2a0 0x7f45ac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:18.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.766+0000 7f45c4eea700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f45ac05c2a0 0x7f45ac05e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f45b40041d0 tx=0x7f45b4009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:18.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.769+0000 7f45b1ffb700 1 -- 172.21.15.46:0/1047790976 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45bc079e60 con 0x7f45c00f56c0 2024-01-21T08:23:18.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.990+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f45c0027a00 con 0x7f45c00f56c0 2024-01-21T08:23:18.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.991+0000 7f45b1ffb700 1 -- 172.21.15.46:0/1047790976 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f45bc04a9f0 con 0x7f45c00f56c0 2024-01-21T08:23:18.992 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:18.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f45ac05c2a0 msgr2=0x7f45ac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:18.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f45ac05c2a0 0x7f45ac05e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f45b40041d0 tx=0x7f45b4009040).stop 2024-01-21T08:23:18.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 msgr2=0x7f45c0106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c0106000 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f45bc009f50 tx=0x7f45bc009b30).stop 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 shutdown_connections 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f45ac05c2a0 0x7f45ac05e760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c00f56c0 0x7f45c0106000 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 --2- 172.21.15.46:0/1047790976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45c00f6960 0x7f45c01064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 >> 172.21.15.46:0/1047790976 conn(0x7f45c00f0e20 msgr2=0x7f45c00fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 shutdown_connections 2024-01-21T08:23:18.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:18.993+0000 7f45c794f700 1 -- 172.21.15.46:0/1047790976 wait complete. 2024-01-21T08:23:19.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:18.154866+0000 mgr.smithi046.lmxnwc (mgr.14164) 604 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:19.345 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:19 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:18.991694+0000 mon.smithi046 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.46:0/1047790976' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:19.385 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:19.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:18.154866+0000 mgr.smithi046.lmxnwc (mgr.14164) 604 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:19.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:19 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:18.991694+0000 mon.smithi046 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.46:0/1047790976' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:20.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:20 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:20.155347+0000 mgr.smithi046.lmxnwc (mgr.14164) 605 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:20.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:20 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:20.155347+0000 mgr.smithi046.lmxnwc (mgr.14164) 605 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:20.765995+0000 mon.smithi046 (mon.0) 592 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:23:21.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:21 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:20.766919+0000 mon.smithi046 (mon.0) 593 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:23:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:20.765995+0000 mon.smithi046 (mon.0) 592 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/mirror_snapshot_schedule"}]: dispatch 2024-01-21T08:23:21.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:21 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:20.766919+0000 mon.smithi046 (mon.0) 593 : audit [INF] from='mgr.14164 172.21.15.46:0/2772010833' entity='mgr.smithi046.lmxnwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi046.lmxnwc/trash_purge_schedule"}]: dispatch 2024-01-21T08:23:22.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:22.635 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:22.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:22 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:22.155857+0000 mgr.smithi046.lmxnwc (mgr.14164) 606 : cluster [DBG] pgmap v567: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:22.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:22 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:22.155857+0000 mgr.smithi046.lmxnwc (mgr.14164) 606 : cluster [DBG] pgmap v567: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 -- 172.21.15.46:0/4042965823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0f9950 msgr2=0x7fb75c0fbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 --2- 172.21.15.46:0/4042965823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0f9950 0x7fb75c0fbda0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fb74c004660 tx=0x7fb74c009b30).stop 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 -- 172.21.15.46:0/4042965823 shutdown_connections 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 --2- 172.21.15.46:0/4042965823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0f9950 0x7fb75c0fbda0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 --2- 172.21.15.46:0/4042965823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb75c0607d0 0x7fb75c0f9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 -- 172.21.15.46:0/4042965823 >> 172.21.15.46:0/4042965823 conn(0x7fb75c0f0e20 msgr2=0x7fb75c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 -- 172.21.15.46:0/4042965823 shutdown_connections 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.612+0000 7fb7638a0700 1 -- 172.21.15.46:0/4042965823 wait complete. 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb7638a0700 1 Processor -- start 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb7638a0700 1 -- start start 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb7638a0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0607d0 0x7fb75c1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb7638a0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb75c0f9950 0x7fb75c107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb7638a0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c107b40 con 0x7fb75c0607d0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb7638a0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c107c80 con 0x7fb75c0f9950 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb76163c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0607d0 0x7fb75c1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb76163c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0607d0 0x7fb75c1070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42336/0 (socket says 172.21.15.46:42336) 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb76163c700 1 -- 172.21.15.46:0/2195126486 learned_addr learned my addr 172.21.15.46:0/2195126486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.613+0000 7fb760e3b700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb75c0f9950 0x7fb75c107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb76163c700 1 -- 172.21.15.46:0/2195126486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb75c0f9950 msgr2=0x7fb75c107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb76163c700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb75c0f9950 0x7fb75c107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb76163c700 1 -- 172.21.15.46:0/2195126486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c005040 con 0x7fb75c0607d0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb76163c700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0607d0 0x7fb75c1070a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fb75800c530 tx=0x7fb758007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb751ffb700 1 -- 172.21.15.46:0/2195126486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb75800f650 con 0x7fb75c0607d0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb751ffb700 1 -- 172.21.15.46:0/2195126486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb75800fc60 con 0x7fb75c0607d0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c108620 con 0x7fb75c0607d0 2024-01-21T08:23:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb751ffb700 1 -- 172.21.15.46:0/2195126486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb758012c30 con 0x7fb75c0607d0 2024-01-21T08:23:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.614+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c108be0 con 0x7fb75c0607d0 2024-01-21T08:23:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.615+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c045ad0 con 0x7fb75c0607d0 2024-01-21T08:23:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.615+0000 7fb751ffb700 1 -- 172.21.15.46:0/2195126486 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb758010070 con 0x7fb75c0607d0 2024-01-21T08:23:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.616+0000 7fb751ffb700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb74805c2a0 0x7fb74805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.616+0000 7fb751ffb700 1 -- 172.21.15.46:0/2195126486 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb7580758b0 con 0x7fb75c0607d0 2024-01-21T08:23:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.616+0000 7fb760e3b700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb74805c2a0 0x7fb74805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.616+0000 7fb760e3b700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb74805c2a0 0x7fb74805e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb74c004630 tx=0x7fb74c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:23.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.620+0000 7fb751ffb700 1 -- 172.21.15.46:0/2195126486 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7580452b0 con 0x7fb75c0607d0 2024-01-21T08:23:23.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.840+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb75c0ff0c0 con 0x7fb75c0607d0 2024-01-21T08:23:23.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.841+0000 7fb751ffb700 1 -- 172.21.15.46:0/2195126486 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fb7580487f0 con 0x7fb75c0607d0 2024-01-21T08:23:23.842 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb74805c2a0 msgr2=0x7fb74805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb74805c2a0 0x7fb74805e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb74c004630 tx=0x7fb74c009b30).stop 2024-01-21T08:23:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0607d0 msgr2=0x7fb75c1070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0607d0 0x7fb75c1070a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fb75800c530 tx=0x7fb758007e20).stop 2024-01-21T08:23:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 shutdown_connections 2024-01-21T08:23:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fb74805c2a0 0x7fb74805e760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb75c0607d0 0x7fb75c1070a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 --2- 172.21.15.46:0/2195126486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb75c0f9950 0x7fb75c107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 >> 172.21.15.46:0/2195126486 conn(0x7fb75c0f0e20 msgr2=0x7fb75c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 shutdown_connections 2024-01-21T08:23:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:23.843+0000 7fb7638a0700 1 -- 172.21.15.46:0/2195126486 wait complete. 2024-01-21T08:23:24.183 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:23 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:23.841764+0000 mon.smithi046 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.46:0/2195126486' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:24.202 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:24.241 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:23 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:23.841764+0000 mon.smithi046 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.46:0/2195126486' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:25.240 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:24 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:24.156252+0000 mgr.smithi046.lmxnwc (mgr.14164) 607 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:25.241 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:24 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:24.156252+0000 mgr.smithi046.lmxnwc (mgr.14164) 607 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:27.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:27.450 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:27.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:27 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:26.156677+0000 mgr.smithi046.lmxnwc (mgr.14164) 608 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:27.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:27 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:26.156677+0000 mgr.smithi046.lmxnwc (mgr.14164) 608 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 -- 172.21.15.46:0/27297083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 msgr2=0x7fad500fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 --2- 172.21.15.46:0/27297083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500fbd70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7fad44004660 tx=0x7fad44009b30).stop 2024-01-21T08:23:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 -- 172.21.15.46:0/27297083 shutdown_connections 2024-01-21T08:23:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 --2- 172.21.15.46:0/27297083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500fbd70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 --2- 172.21.15.46:0/27297083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad50060870 0x7fad500f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 -- 172.21.15.46:0/27297083 >> 172.21.15.46:0/27297083 conn(0x7fad500f0e20 msgr2=0x7fad500f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:28.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 -- 172.21.15.46:0/27297083 shutdown_connections 2024-01-21T08:23:28.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.680+0000 7fad561df700 1 -- 172.21.15.46:0/27297083 wait complete. 2024-01-21T08:23:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad561df700 1 Processor -- start 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad561df700 1 -- start start 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad561df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad50060870 0x7fad500f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad561df700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad561df700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad500f5800 con 0x7fad500f9920 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad561df700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad500f5940 con 0x7fad50060870 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad4effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.681+0000 7fad4effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42358/0 (socket says 172.21.15.46:42358) 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad4effd700 1 -- 172.21.15.46:0/1324406977 learned_addr learned my addr 172.21.15.46:0/1324406977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad4f7fe700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad50060870 0x7fad500f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad4effd700 1 -- 172.21.15.46:0/1324406977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad50060870 msgr2=0x7fad500f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad4effd700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad50060870 0x7fad500f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad4effd700 1 -- 172.21.15.46:0/1324406977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44005040 con 0x7fad500f9920 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad4effd700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500f52a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fad44000c00 tx=0x7fad4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad37fff700 1 -- 172.21.15.46:0/1324406977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad44019070 con 0x7fad500f9920 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad37fff700 1 -- 172.21.15.46:0/1324406977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad44007e70 con 0x7fad500f9920 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad500f5bc0 con 0x7fad500f9920 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad37fff700 1 -- 172.21.15.46:0/1324406977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad44014d40 con 0x7fad500f9920 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.682+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad500f6120 con 0x7fad500f9920 2024-01-21T08:23:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.683+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad50045ad0 con 0x7fad500f9920 2024-01-21T08:23:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.684+0000 7fad37fff700 1 -- 172.21.15.46:0/1324406977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fad44025070 con 0x7fad500f9920 2024-01-21T08:23:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.684+0000 7fad37fff700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad3805c2a0 0x7fad3805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.684+0000 7fad37fff700 1 -- 172.21.15.46:0/1324406977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fad44076a20 con 0x7fad500f9920 2024-01-21T08:23:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.684+0000 7fad4f7fe700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad3805c2a0 0x7fad3805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.684+0000 7fad4f7fe700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad3805c2a0 0x7fad3805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fad500f6bb0 tx=0x7fad4000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:28.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.688+0000 7fad37fff700 1 -- 172.21.15.46:0/1324406977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad44049aa0 con 0x7fad500f9920 2024-01-21T08:23:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.907+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fad50027a00 con 0x7fad500f9920 2024-01-21T08:23:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.907+0000 7fad37fff700 1 -- 172.21.15.46:0/1324406977 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fad4404d0c0 con 0x7fad500f9920 2024-01-21T08:23:28.909 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad3805c2a0 msgr2=0x7fad3805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad3805c2a0 0x7fad3805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fad500f6bb0 tx=0x7fad4000b040).stop 2024-01-21T08:23:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 msgr2=0x7fad500f52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500f52a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fad44000c00 tx=0x7fad4400b040).stop 2024-01-21T08:23:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 shutdown_connections 2024-01-21T08:23:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fad3805c2a0 0x7fad3805e760 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad50060870 0x7fad500f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 --2- 172.21.15.46:0/1324406977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad500f9920 0x7fad500f52a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 >> 172.21.15.46:0/1324406977 conn(0x7fad500f0e20 msgr2=0x7fad500f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 shutdown_connections 2024-01-21T08:23:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:28.909+0000 7fad561df700 1 -- 172.21.15.46:0/1324406977 wait complete. 2024-01-21T08:23:29.468 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:28.157012+0000 mgr.smithi046.lmxnwc (mgr.14164) 609 : cluster [DBG] pgmap v570: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:29.468 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:29 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:28.907858+0000 mon.smithi046 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.46:0/1324406977' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:29.469 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:29.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:28.157012+0000 mgr.smithi046.lmxnwc (mgr.14164) 609 : cluster [DBG] pgmap v570: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:29.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:29 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:28.907858+0000 mon.smithi046 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.46:0/1324406977' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:30.740 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:30 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:30.157338+0000 mgr.smithi046.lmxnwc (mgr.14164) 610 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:30.741 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:30 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:30.157338+0000 mgr.smithi046.lmxnwc (mgr.14164) 610 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:32.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:32.717 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:33.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:33 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:32.157773+0000 mgr.smithi046.lmxnwc (mgr.14164) 611 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:33.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:33 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:32.157773+0000 mgr.smithi046.lmxnwc (mgr.14164) 611 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 -- 172.21.15.46:0/1313575587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f7a80 msgr2=0x7f89380f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 --2- 172.21.15.46:0/1313575587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f7a80 0x7f89380f7ee0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f892c004660 tx=0x7f892c009b30).stop 2024-01-21T08:23:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 -- 172.21.15.46:0/1313575587 shutdown_connections 2024-01-21T08:23:33.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 --2- 172.21.15.46:0/1313575587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f7a80 0x7f89380f7ee0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 --2- 172.21.15.46:0/1313575587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89380f67e0 0x7f89380f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 -- 172.21.15.46:0/1313575587 >> 172.21.15.46:0/1313575587 conn(0x7f89380f1f80 msgr2=0x7f89380f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 -- 172.21.15.46:0/1313575587 shutdown_connections 2024-01-21T08:23:33.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.783+0000 7f893d5e1700 1 -- 172.21.15.46:0/1313575587 wait complete. 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.784+0000 7f893d5e1700 1 Processor -- start 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.784+0000 7f893d5e1700 1 -- start start 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.784+0000 7f893d5e1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f67e0 0x7f8938106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.784+0000 7f893d5e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89380f7a80 0x7f89381074b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.784+0000 7f893d5e1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8938107a60 con 0x7f89380f67e0 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.784+0000 7f893d5e1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8938107ba0 con 0x7f89380f7a80 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f8936ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f67e0 0x7f8938106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f8936ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f67e0 0x7f8938106fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53772/0 (socket says 172.21.15.46:53772) 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f8936ffd700 1 -- 172.21.15.46:0/2858314385 learned_addr learned my addr 172.21.15.46:0/2858314385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f89367fc700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89380f7a80 0x7f89381074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:33.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f8936ffd700 1 -- 172.21.15.46:0/2858314385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89380f7a80 msgr2=0x7f89381074b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f8936ffd700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89380f7a80 0x7f89381074b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f8936ffd700 1 -- 172.21.15.46:0/2858314385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c005040 con 0x7f89380f67e0 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f8936ffd700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f67e0 0x7f8938106fc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f892800e7c0 tx=0x7f892800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f891f7fe700 1 -- 172.21.15.46:0/2858314385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89280092e0 con 0x7f89380f67e0 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f891f7fe700 1 -- 172.21.15.46:0/2858314385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f892800f040 con 0x7f89380f67e0 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8938108540 con 0x7f89380f67e0 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f891f7fe700 1 -- 172.21.15.46:0/2858314385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89280147c0 con 0x7f89380f67e0 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.785+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8938108b00 con 0x7f89380f67e0 2024-01-21T08:23:33.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.786+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8938045ad0 con 0x7f89380f67e0 2024-01-21T08:23:33.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.787+0000 7f891f7fe700 1 -- 172.21.15.46:0/2858314385 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f892801b030 con 0x7f89380f67e0 2024-01-21T08:23:33.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.787+0000 7f891f7fe700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f892005c2a0 0x7f892005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:33.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.787+0000 7f891f7fe700 1 -- 172.21.15.46:0/2858314385 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8928075a90 con 0x7f89380f67e0 2024-01-21T08:23:33.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.787+0000 7f89367fc700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f892005c2a0 0x7f892005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:33.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.787+0000 7f89367fc700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f892005c2a0 0x7f892005e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f892c005010 tx=0x7f892c016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:33.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:33.791+0000 7f891f7fe700 1 -- 172.21.15.46:0/2858314385 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8928048a90 con 0x7f89380f67e0 2024-01-21T08:23:34.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.008+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8938027a00 con 0x7f89380f67e0 2024-01-21T08:23:34.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.009+0000 7f891f7fe700 1 -- 172.21.15.46:0/2858314385 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f8928019020 con 0x7f89380f67e0 2024-01-21T08:23:34.010 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f892005c2a0 msgr2=0x7f892005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f892005c2a0 0x7f892005e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f892c005010 tx=0x7f892c016040).stop 2024-01-21T08:23:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f67e0 msgr2=0x7f8938106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f67e0 0x7f8938106fc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f892800e7c0 tx=0x7f892800b040).stop 2024-01-21T08:23:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 shutdown_connections 2024-01-21T08:23:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f892005c2a0 0x7f892005e760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89380f67e0 0x7f8938106fc0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 --2- 172.21.15.46:0/2858314385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89380f7a80 0x7f89381074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 >> 172.21.15.46:0/2858314385 conn(0x7f89380f1f80 msgr2=0x7f89380fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 shutdown_connections 2024-01-21T08:23:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:34.010+0000 7f893d5e1700 1 -- 172.21.15.46:0/2858314385 wait complete. 2024-01-21T08:23:34.440 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:34 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:34.009493+0000 mon.smithi046 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.46:0/2858314385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:34.460 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:34.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:34 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:34.009493+0000 mon.smithi046 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.46:0/2858314385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:35.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:35 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:34.158263+0000 mgr.smithi046.lmxnwc (mgr.14164) 612 : cluster [DBG] pgmap v573: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:35.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:35 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:34.158263+0000 mgr.smithi046.lmxnwc (mgr.14164) 612 : cluster [DBG] pgmap v573: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:37.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:37.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:37 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:36.158762+0000 mgr.smithi046.lmxnwc (mgr.14164) 613 : cluster [DBG] pgmap v574: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:37.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:37 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:36.158762+0000 mgr.smithi046.lmxnwc (mgr.14164) 613 : cluster [DBG] pgmap v574: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:38.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 -- 172.21.15.46:0/4156009302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f6960 msgr2=0x7f48e00f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:38.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 --2- 172.21.15.46:0/4156009302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f6960 0x7f48e00f6dc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f48d0004660 tx=0x7f48d0009b30).stop 2024-01-21T08:23:38.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 -- 172.21.15.46:0/4156009302 shutdown_connections 2024-01-21T08:23:38.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 --2- 172.21.15.46:0/4156009302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f6960 0x7f48e00f6dc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 --2- 172.21.15.46:0/4156009302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48e00f56c0 0x7f48e00f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 -- 172.21.15.46:0/4156009302 >> 172.21.15.46:0/4156009302 conn(0x7f48e00f0e20 msgr2=0x7f48e00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 -- 172.21.15.46:0/4156009302 shutdown_connections 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 -- 172.21.15.46:0/4156009302 wait complete. 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.766+0000 7f48e8012700 1 Processor -- start 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e8012700 1 -- start start 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e8012700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f56c0 0x7f48e0109de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e8012700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48e00f6960 0x7f48e010c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e8012700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0060940 con 0x7f48e00f56c0 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e8012700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0060a80 con 0x7f48e00f6960 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e5dae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f56c0 0x7f48e0109de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e5dae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f56c0 0x7f48e0109de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53786/0 (socket says 172.21.15.46:53786) 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e5dae700 1 -- 172.21.15.46:0/480348701 learned_addr learned my addr 172.21.15.46:0/480348701 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e55ad700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48e00f6960 0x7f48e010c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e5dae700 1 -- 172.21.15.46:0/480348701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48e00f6960 msgr2=0x7f48e010c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:38.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e5dae700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48e00f6960 0x7f48e010c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e5dae700 1 -- 172.21.15.46:0/480348701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d0005040 con 0x7f48e00f56c0 2024-01-21T08:23:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.767+0000 7f48e5dae700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f56c0 0x7f48e0109de0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f48dc00c790 tx=0x7f48dc007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.768+0000 7f48d67fc700 1 -- 172.21.15.46:0/480348701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48dc00e680 con 0x7f48e00f56c0 2024-01-21T08:23:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.768+0000 7f48d67fc700 1 -- 172.21.15.46:0/480348701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48dc00ec90 con 0x7f48e00f56c0 2024-01-21T08:23:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.768+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e010c890 con 0x7f48e00f56c0 2024-01-21T08:23:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.768+0000 7f48d67fc700 1 -- 172.21.15.46:0/480348701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48dc012cb0 con 0x7f48e00f56c0 2024-01-21T08:23:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.768+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e010ce50 con 0x7f48e00f56c0 2024-01-21T08:23:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.769+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e0045ad0 con 0x7f48e00f56c0 2024-01-21T08:23:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.769+0000 7f48d67fc700 1 -- 172.21.15.46:0/480348701 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f48dc010070 con 0x7f48e00f56c0 2024-01-21T08:23:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.769+0000 7f48d67fc700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f48cc05c2a0 0x7f48cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.769+0000 7f48d67fc700 1 -- 172.21.15.46:0/480348701 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f48dc075c90 con 0x7f48e00f56c0 2024-01-21T08:23:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.769+0000 7f48e55ad700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f48cc05c2a0 0x7f48cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.770+0000 7f48e55ad700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f48cc05c2a0 0x7f48cc05e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f48d0013580 tx=0x7f48d0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.773+0000 7f48d67fc700 1 -- 172.21.15.46:0/480348701 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48dc048c90 con 0x7f48e00f56c0 2024-01-21T08:23:38.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.992+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f48e0027a00 con 0x7f48e00f56c0 2024-01-21T08:23:38.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.992+0000 7f48d67fc700 1 -- 172.21.15.46:0/480348701 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f48dc04c2b0 con 0x7f48e00f56c0 2024-01-21T08:23:38.994 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f48cc05c2a0 msgr2=0x7f48cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f48cc05c2a0 0x7f48cc05e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f48d0013580 tx=0x7f48d0009b30).stop 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f56c0 msgr2=0x7f48e0109de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f56c0 0x7f48e0109de0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f48dc00c790 tx=0x7f48dc007d90).stop 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 shutdown_connections 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f48cc05c2a0 0x7f48cc05e760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48e00f56c0 0x7f48e0109de0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 --2- 172.21.15.46:0/480348701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48e00f6960 0x7f48e010c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 >> 172.21.15.46:0/480348701 conn(0x7f48e00f0e20 msgr2=0x7f48e00fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 shutdown_connections 2024-01-21T08:23:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:38.994+0000 7f48e8012700 1 -- 172.21.15.46:0/480348701 wait complete. 2024-01-21T08:23:39.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:38.159105+0000 mgr.smithi046.lmxnwc (mgr.14164) 614 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:39.306 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:39 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:38.993037+0000 mon.smithi046 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.46:0/480348701' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:39.373 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:39.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:38.159105+0000 mgr.smithi046.lmxnwc (mgr.14164) 614 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:39.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:39 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:38.993037+0000 mon.smithi046 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.46:0/480348701' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:40.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:40 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:40.159636+0000 mgr.smithi046.lmxnwc (mgr.14164) 615 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:40.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:40 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:40.159636+0000 mgr.smithi046.lmxnwc (mgr.14164) 615 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:42.374 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:43.428 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:43 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:42.160051+0000 mgr.smithi046.lmxnwc (mgr.14164) 616 : cluster [DBG] pgmap v577: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:43.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:43 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:42.160051+0000 mgr.smithi046.lmxnwc (mgr.14164) 616 : cluster [DBG] pgmap v577: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:43.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.633+0000 7f409cd45700 1 -- 172.21.15.46:0/3605743299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 msgr2=0x7f40980f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.633+0000 7f409cd45700 1 --2- 172.21.15.46:0/3605743299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f40980f5ae0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f4088004660 tx=0x7f4088009b30).stop 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.633+0000 7f409cd45700 1 -- 172.21.15.46:0/3605743299 shutdown_connections 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.633+0000 7f409cd45700 1 --2- 172.21.15.46:0/3605743299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40980f6960 0x7f40980f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.633+0000 7f409cd45700 1 --2- 172.21.15.46:0/3605743299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f40980f5ae0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.633+0000 7f409cd45700 1 -- 172.21.15.46:0/3605743299 >> 172.21.15.46:0/3605743299 conn(0x7f40980f0e20 msgr2=0x7f40980f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.633+0000 7f409cd45700 1 -- 172.21.15.46:0/3605743299 shutdown_connections 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409cd45700 1 -- 172.21.15.46:0/3605743299 wait complete. 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409cd45700 1 Processor -- start 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409cd45700 1 -- start start 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409cd45700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f4098109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:43.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409cd45700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40980f6960 0x7f409810c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:43.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409cd45700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4098060920 con 0x7f40980f56c0 2024-01-21T08:23:43.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409cd45700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4098060a60 con 0x7f40980f6960 2024-01-21T08:23:43.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f4098109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:43.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f4098109df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37460/0 (socket says 172.21.15.46:37460) 2024-01-21T08:23:43.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.634+0000 7f409659c700 1 -- 172.21.15.46:0/222891355 learned_addr learned my addr 172.21.15.46:0/222891355 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:43.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f4095d9b700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40980f6960 0x7f409810c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f409659c700 1 -- 172.21.15.46:0/222891355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40980f6960 msgr2=0x7f409810c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f409659c700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40980f6960 0x7f409810c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f409659c700 1 -- 172.21.15.46:0/222891355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4088005040 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f4095d9b700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40980f6960 0x7f409810c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f409659c700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f4098109df0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f4088009f50 tx=0x7f4088009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f408effd700 1 -- 172.21.15.46:0/222891355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4088018070 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409810c840 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f408effd700 1 -- 172.21.15.46:0/222891355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4088013410 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f408effd700 1 -- 172.21.15.46:0/222891355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f408801c620 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.635+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409810cd70 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.636+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4098059100 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.636+0000 7f408effd700 1 -- 172.21.15.46:0/222891355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f408801d420 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.637+0000 7f408effd700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f408405c2a0 0x7f408405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.637+0000 7f408effd700 1 -- 172.21.15.46:0/222891355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4088077a20 con 0x7f40980f56c0 2024-01-21T08:23:43.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.637+0000 7f4095d9b700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f408405c2a0 0x7f408405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:43.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.637+0000 7f4095d9b700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f408405c2a0 0x7f408405e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f4080004a70 tx=0x7f4080009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:43.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.641+0000 7f408effd700 1 -- 172.21.15.46:0/222891355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f408804aa20 con 0x7f40980f56c0 2024-01-21T08:23:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.861+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4098045ad0 con 0x7f40980f56c0 2024-01-21T08:23:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.862+0000 7f408effd700 1 -- 172.21.15.46:0/222891355 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f408804e040 con 0x7f40980f56c0 2024-01-21T08:23:43.864 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.863+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f408405c2a0 msgr2=0x7f408405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.863+0000 7f409cd45700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f408405c2a0 0x7f408405e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f4080004a70 tx=0x7f4080009040).stop 2024-01-21T08:23:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.863+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 msgr2=0x7f4098109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.863+0000 7f409cd45700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f4098109df0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f4088009f50 tx=0x7f4088009b30).stop 2024-01-21T08:23:43.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.864+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 shutdown_connections 2024-01-21T08:23:43.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.864+0000 7f409cd45700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f408405c2a0 0x7f408405e760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:43.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.864+0000 7f409cd45700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980f56c0 0x7f4098109df0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:43.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.864+0000 7f409cd45700 1 --2- 172.21.15.46:0/222891355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40980f6960 0x7f409810c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:43.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.864+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 >> 172.21.15.46:0/222891355 conn(0x7f40980f0e20 msgr2=0x7f40980fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:43.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.864+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 shutdown_connections 2024-01-21T08:23:43.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:43.864+0000 7f409cd45700 1 -- 172.21.15.46:0/222891355 wait complete. 2024-01-21T08:23:44.192 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:44.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:44 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:43.862854+0000 mon.smithi046 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.46:0/222891355' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:44.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:44 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:43.862854+0000 mon.smithi046 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.46:0/222891355' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:45.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:45 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:44.160520+0000 mgr.smithi046.lmxnwc (mgr.14164) 617 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:45.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:45 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:44.160520+0000 mgr.smithi046.lmxnwc (mgr.14164) 617 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:47.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:47.441 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:47.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:47 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:46.160980+0000 mgr.smithi046.lmxnwc (mgr.14164) 618 : cluster [DBG] pgmap v579: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:47.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:47 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:46.160980+0000 mgr.smithi046.lmxnwc (mgr.14164) 618 : cluster [DBG] pgmap v579: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:48.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 -- 172.21.15.46:0/448661023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 msgr2=0x7fc19c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 --2- 172.21.15.46:0/448661023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c0f6dc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fc190004660 tx=0x7fc190009b30).stop 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 -- 172.21.15.46:0/448661023 shutdown_connections 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 --2- 172.21.15.46:0/448661023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c0f6dc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 --2- 172.21.15.46:0/448661023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19c0f56c0 0x7fc19c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 -- 172.21.15.46:0/448661023 >> 172.21.15.46:0/448661023 conn(0x7fc19c0f0e20 msgr2=0x7fc19c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 -- 172.21.15.46:0/448661023 shutdown_connections 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.385+0000 7fc1a4013700 1 -- 172.21.15.46:0/448661023 wait complete. 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a4013700 1 Processor -- start 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a4013700 1 -- start start 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a4013700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19c0f56c0 0x7fc19c1070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a4013700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c1075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a4013700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c107b70 con 0x7fc19c0f6960 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a4013700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c107cb0 con 0x7fc19c0f56c0 2024-01-21T08:23:48.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a15ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c1075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a15ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c1075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37474/0 (socket says 172.21.15.46:37474) 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a15ae700 1 -- 172.21.15.46:0/2631656347 learned_addr learned my addr 172.21.15.46:0/2631656347 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a1daf700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19c0f56c0 0x7fc19c1070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a15ae700 1 -- 172.21.15.46:0/2631656347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19c0f56c0 msgr2=0x7fc19c1070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a15ae700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19c0f56c0 0x7fc19c1070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a15ae700 1 -- 172.21.15.46:0/2631656347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc190005040 con 0x7fc19c0f6960 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.386+0000 7fc1a15ae700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c1075c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fc190009830 tx=0x7fc190009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.387+0000 7fc18e7fc700 1 -- 172.21.15.46:0/2631656347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc190018070 con 0x7fc19c0f6960 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.387+0000 7fc18e7fc700 1 -- 172.21.15.46:0/2631656347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc190013410 con 0x7fc19c0f6960 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.387+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c1085f0 con 0x7fc19c0f6960 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.387+0000 7fc18e7fc700 1 -- 172.21.15.46:0/2631656347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19001c400 con 0x7fc19c0f6960 2024-01-21T08:23:48.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.387+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c108b50 con 0x7fc19c0f6960 2024-01-21T08:23:48.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.388+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c045ad0 con 0x7fc19c0f6960 2024-01-21T08:23:48.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.388+0000 7fc18e7fc700 1 -- 172.21.15.46:0/2631656347 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc19001d420 con 0x7fc19c0f6960 2024-01-21T08:23:48.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.389+0000 7fc18e7fc700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc18805c2a0 0x7fc18805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:48.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.389+0000 7fc18e7fc700 1 -- 172.21.15.46:0/2631656347 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc190077930 con 0x7fc19c0f6960 2024-01-21T08:23:48.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.389+0000 7fc1a1daf700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc18805c2a0 0x7fc18805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:48.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.389+0000 7fc1a1daf700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc18805c2a0 0x7fc18805e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fc19c0f67c0 tx=0x7fc198009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:48.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.393+0000 7fc18e7fc700 1 -- 172.21.15.46:0/2631656347 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc19004a9b0 con 0x7fc19c0f6960 2024-01-21T08:23:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.612+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc19c0fdf00 con 0x7fc19c0f6960 2024-01-21T08:23:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.612+0000 7fc18e7fc700 1 -- 172.21.15.46:0/2631656347 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fc19004dfd0 con 0x7fc19c0f6960 2024-01-21T08:23:48.614 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc18805c2a0 msgr2=0x7fc18805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc18805c2a0 0x7fc18805e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fc19c0f67c0 tx=0x7fc198009040).stop 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 msgr2=0x7fc19c1075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c1075c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fc190009830 tx=0x7fc190009b30).stop 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 shutdown_connections 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7fc18805c2a0 0x7fc18805e760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19c0f56c0 0x7fc19c1070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 --2- 172.21.15.46:0/2631656347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19c0f6960 0x7fc19c1075c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 >> 172.21.15.46:0/2631656347 conn(0x7fc19c0f0e20 msgr2=0x7fc19c0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 shutdown_connections 2024-01-21T08:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:48.614+0000 7fc1a4013700 1 -- 172.21.15.46:0/2631656347 wait complete. 2024-01-21T08:23:48.989 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:49.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:48.161385+0000 mgr.smithi046.lmxnwc (mgr.14164) 619 : cluster [DBG] pgmap v580: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:49.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:49 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:48.613238+0000 mon.smithi046 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.46:0/2631656347' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:49.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:48.161385+0000 mgr.smithi046.lmxnwc (mgr.14164) 619 : cluster [DBG] pgmap v580: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:49.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:49 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:48.613238+0000 mon.smithi046 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.46:0/2631656347' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:50.990 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:50 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:50.161843+0000 mgr.smithi046.lmxnwc (mgr.14164) 620 : cluster [DBG] pgmap v581: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:50.991 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:50 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:50.161843+0000 mgr.smithi046.lmxnwc (mgr.14164) 620 : cluster [DBG] pgmap v581: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:51.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.235+0000 7f606300b700 1 -- 172.21.15.46:0/161243775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 msgr2=0x7f605c0f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.235+0000 7f606300b700 1 --2- 172.21.15.46:0/161243775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c0f5930 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f6058004660 tx=0x7f6058009b30).stop 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 -- 172.21.15.46:0/161243775 shutdown_connections 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 --2- 172.21.15.46:0/161243775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f605c0f67b0 0x7f605c0f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 --2- 172.21.15.46:0/161243775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c0f5930 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 -- 172.21.15.46:0/161243775 >> 172.21.15.46:0/161243775 conn(0x7f605c0f0c70 msgr2=0x7f605c0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 -- 172.21.15.46:0/161243775 shutdown_connections 2024-01-21T08:23:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 -- 172.21.15.46:0/161243775 wait complete. 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 Processor -- start 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 -- start start 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c105ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f605c0f67b0 0x7f605c106390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c106940 con 0x7f605c0f5510 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.236+0000 7f606300b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c107140 con 0x7f605c0f67b0 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6062009700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c105ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6062009700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c105ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54618/0 (socket says 172.21.15.46:54618) 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6062009700 1 -- 172.21.15.46:0/2357761108 learned_addr learned my addr 172.21.15.46:0/2357761108 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6061808700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f605c0f67b0 0x7f605c106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6062009700 1 -- 172.21.15.46:0/2357761108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f605c0f67b0 msgr2=0x7f605c106390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6062009700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f605c0f67b0 0x7f605c106390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6062009700 1 -- 172.21.15.46:0/2357761108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6058005040 con 0x7f605c0f5510 2024-01-21T08:23:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6062009700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c105ea0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f6058009f50 tx=0x7f6058009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6052ffd700 1 -- 172.21.15.46:0/2357761108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6058018070 con 0x7f605c0f5510 2024-01-21T08:23:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6052ffd700 1 -- 172.21.15.46:0/2357761108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6058013410 con 0x7f605c0f5510 2024-01-21T08:23:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c1073c0 con 0x7f605c0f5510 2024-01-21T08:23:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f6052ffd700 1 -- 172.21.15.46:0/2357761108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f605801c620 con 0x7f605c0f5510 2024-01-21T08:23:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.237+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c107920 con 0x7f605c0f5510 2024-01-21T08:23:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.238+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c059100 con 0x7f605c0f5510 2024-01-21T08:23:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.239+0000 7f6052ffd700 1 -- 172.21.15.46:0/2357761108 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f605801d420 con 0x7f605c0f5510 2024-01-21T08:23:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.239+0000 7f6052ffd700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f604805c2b0 0x7f604805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.239+0000 7f6052ffd700 1 -- 172.21.15.46:0/2357761108 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6058077f00 con 0x7f605c0f5510 2024-01-21T08:23:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.239+0000 7f6061808700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f604805c2b0 0x7f604805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.240+0000 7f6061808700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f604805c2b0 0x7f604805e770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f604c007fd0 tx=0x7f604c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.243+0000 7f6052ffd700 1 -- 172.21.15.46:0/2357761108 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f605804af30 con 0x7f605c0f5510 2024-01-21T08:23:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.461+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f605c045ad0 con 0x7f605c0f5510 2024-01-21T08:23:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.462+0000 7f6052ffd700 1 -- 172.21.15.46:0/2357761108 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f605c045ad0 con 0x7f605c0f5510 2024-01-21T08:23:53.463 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f604805c2b0 msgr2=0x7f604805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f604805c2b0 0x7f604805e770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f604c007fd0 tx=0x7f604c009040).stop 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 msgr2=0x7f605c105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c105ea0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f6058009f50 tx=0x7f6058009b30).stop 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 shutdown_connections 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f604805c2b0 0x7f604805e770 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f605c0f5510 0x7f605c105ea0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 --2- 172.21.15.46:0/2357761108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f605c0f67b0 0x7f605c106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 >> 172.21.15.46:0/2357761108 conn(0x7f605c0f0c70 msgr2=0x7f605c0fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 shutdown_connections 2024-01-21T08:23:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:53.463+0000 7f606300b700 1 -- 172.21.15.46:0/2357761108 wait complete. 2024-01-21T08:23:53.488 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:53 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:52.162142+0000 mgr.smithi046.lmxnwc (mgr.14164) 621 : cluster [DBG] pgmap v582: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:53.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:53 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:52.162142+0000 mgr.smithi046.lmxnwc (mgr.14164) 621 : cluster [DBG] pgmap v582: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:53.864 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:54.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:54 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: audit 2024-01-21T08:23:53.462609+0000 mon.smithi046 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.46:0/2357761108' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:54.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:54 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: audit 2024-01-21T08:23:53.462609+0000 mon.smithi046 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.46:0/2357761108' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-21T08:23:55.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:55 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:54.162541+0000 mgr.smithi046.lmxnwc (mgr.14164) 622 : cluster [DBG] pgmap v583: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:55.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:55 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:54.162541+0000 mgr.smithi046.lmxnwc (mgr.14164) 622 : cluster [DBG] pgmap v583: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:56.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid a17ce476-b833-11ee-95b0-87774f69a715 -- ceph health --format=json 2024-01-21T08:23:57.110 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/mon.smithi046/config 2024-01-21T08:23:57.490 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:57 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: cluster 2024-01-21T08:23:56.163044+0000 mgr.smithi046.lmxnwc (mgr.14164) 623 : cluster [DBG] pgmap v584: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:57.491 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:57 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: cluster 2024-01-21T08:23:56.163044+0000 mgr.smithi046.lmxnwc (mgr.14164) 623 : cluster [DBG] pgmap v584: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-21T08:23:58.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.049+0000 7f127e5ba700 1 -- 172.21.15.46:0/1328906884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 msgr2=0x7f12780fbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:58.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.049+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1328906884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12780fbbd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f1268004660 tx=0x7f1268009b30).stop 2024-01-21T08:23:58.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.049+0000 7f127e5ba700 1 -- 172.21.15.46:0/1328906884 shutdown_connections 2024-01-21T08:23:58.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.049+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1328906884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12780fbbd0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:58.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.049+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1328906884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12780607b0 0x7f12780f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:58.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.049+0000 7f127e5ba700 1 -- 172.21.15.46:0/1328906884 >> 172.21.15.46:0/1328906884 conn(0x7f12780f0c70 msgr2=0x7f12780f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:58.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.050+0000 7f127e5ba700 1 -- 172.21.15.46:0/1328906884 shutdown_connections 2024-01-21T08:23:58.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.050+0000 7f127e5ba700 1 -- 172.21.15.46:0/1328906884 wait complete. 2024-01-21T08:23:58.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.050+0000 7f127e5ba700 1 Processor -- start 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127e5ba700 1 -- start start 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127e5ba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12780607b0 0x7f1278106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127e5ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12781073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127e5ba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278107980 con 0x7f12780f97a0 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127e5ba700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278107ac0 con 0x7f12780607b0 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127cdb7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12781073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127cdb7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12781073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54634/0 (socket says 172.21.15.46:54634) 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127cdb7700 1 -- 172.21.15.46:0/1590870925 learned_addr learned my addr 172.21.15.46:0/1590870925 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127d5b8700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12780607b0 0x7f1278106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127cdb7700 1 -- 172.21.15.46:0/1590870925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12780607b0 msgr2=0x7f1278106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127cdb7700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12780607b0 0x7f1278106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127cdb7700 1 -- 172.21.15.46:0/1590870925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268005040 con 0x7f12780f97a0 2024-01-21T08:23:58.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127d5b8700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12780607b0 0x7f1278106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-21T08:23:58.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127cdb7700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12781073d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f1268009830 tx=0x7f1268009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:58.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f126dffb700 1 -- 172.21.15.46:0/1590870925 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1268018070 con 0x7f12780f97a0 2024-01-21T08:23:58.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.051+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1278108400 con 0x7f12780f97a0 2024-01-21T08:23:58.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.052+0000 7f126dffb700 1 -- 172.21.15.46:0/1590870925 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1268013410 con 0x7f12780f97a0 2024-01-21T08:23:58.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.052+0000 7f126dffb700 1 -- 172.21.15.46:0/1590870925 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126801c440 con 0x7f12780f97a0 2024-01-21T08:23:58.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.052+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1278108960 con 0x7f12780f97a0 2024-01-21T08:23:58.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.053+0000 7f126dffb700 1 -- 172.21.15.46:0/1590870925 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f126801c6c0 con 0x7f12780f97a0 2024-01-21T08:23:58.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.053+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1278045ad0 con 0x7f12780f97a0 2024-01-21T08:23:58.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.053+0000 7f126dffb700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f126405c2b0 0x7f126405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T08:23:58.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.053+0000 7f126dffb700 1 -- 172.21.15.46:0/1590870925 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1268077a30 con 0x7f12780f97a0 2024-01-21T08:23:58.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.053+0000 7f127d5b8700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f126405c2b0 0x7f126405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T08:23:58.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.054+0000 7f127d5b8700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f126405c2b0 0x7f126405e770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f12740041d0 tx=0x7f1274009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T08:23:58.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.057+0000 7f126dffb700 1 -- 172.21.15.46:0/1590870925 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f126804aa30 con 0x7f12780f97a0 2024-01-21T08:23:58.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.276+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1278027a00 con 0x7f12780f97a0 2024-01-21T08:23:58.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.277+0000 7f126dffb700 1 -- 172.21.15.46:0/1590870925 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f126804e050 con 0x7f12780f97a0 2024-01-21T08:23:58.278 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-21T08:23:58.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.278+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f126405c2b0 msgr2=0x7f126405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:58.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.278+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f126405c2b0 0x7f126405e770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f12740041d0 tx=0x7f1274009040).stop 2024-01-21T08:23:58.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.278+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 msgr2=0x7f12781073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T08:23:58.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.278+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12781073d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f1268009830 tx=0x7f1268009b30).stop 2024-01-21T08:23:58.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.279+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 shutdown_connections 2024-01-21T08:23:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.279+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:6800/559524926,v1:172.21.15.46:6801/559524926] conn(0x7f126405c2b0 0x7f126405e770 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.279+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12780607b0 0x7f1278106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.279+0000 7f127e5ba700 1 --2- 172.21.15.46:0/1590870925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12780f97a0 0x7f12781073d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T08:23:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.279+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 >> 172.21.15.46:0/1590870925 conn(0x7f12780f0c70 msgr2=0x7f12780f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T08:23:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.279+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 shutdown_connections 2024-01-21T08:23:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-21T08:23:58.279+0000 7f127e5ba700 1 -- 172.21.15.46:0/1590870925 wait complete. 2024-01-21T08:23:58.647 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"2 failed cephadm daemon(s)","count":2},"muted":false}},"mutes":[]} 2024-01-21T08:23:58.647 INFO:tasks.cephadm:Teardown begin 2024-01-21T08:23:58.647 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-01-21T08:23:58.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-21T08:23:58.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-21T08:23:58.720 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-21T08:23:58.720 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T08:23:58.738 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T08:23:58.782 INFO:tasks.cephadm:Stopping all daemons... 2024-01-21T08:23:58.783 INFO:tasks.cephadm.mon.smithi046:Stopping mon.smithi046... 2024-01-21T08:23:58.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046 2024-01-21T08:23:58.899 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:58 smithi046 systemd[1]: Stopping Ceph mon.smithi046 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:23:59.226 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:23:58.897+0000 7f597c988700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi046 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T08:23:59.226 INFO:journalctl@ceph.mon.smithi046.smithi046.stdout:Jan 21 08:23:58 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi046[38805]: debug 2024-01-21T08:23:58.897+0000 7f597c988700 -1 mon.smithi046@0(leader) e2 *** Got Signal Terminated *** 2024-01-21T08:23:59.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi046.service' 2024-01-21T08:23:59.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:23:59.468 INFO:tasks.cephadm.mon.smithi046:Stopped mon.smithi046 2024-01-21T08:23:59.468 INFO:tasks.cephadm.mon.smithi078:Stopping mon.smithi078... 2024-01-21T08:23:59.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi078 2024-01-21T08:23:59.788 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:59 smithi078 systemd[1]: Stopping Ceph mon.smithi078 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:23:59.788 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:23:59.578+0000 7f36bd3c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi078 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T08:23:59.788 INFO:journalctl@ceph.mon.smithi078.smithi078.stdout:Jan 21 08:23:59 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-mon.smithi078[57637]: debug 2024-01-21T08:23:59.578+0000 7f36bd3c2700 -1 mon.smithi078@1(peon) e2 *** Got Signal Terminated *** 2024-01-21T08:23:59.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@mon.smithi078.service' 2024-01-21T08:23:59.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:23:59.929 INFO:tasks.cephadm.mon.smithi078:Stopped mon.smithi078 2024-01-21T08:23:59.929 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-01-21T08:23:59.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.0 2024-01-21T08:24:00.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:23:59 smithi046 systemd[1]: Stopping Ceph osd.0 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:24:00.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:24:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0[47710]: debug 2024-01-21T08:24:00.048+0000 7fc09008e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-21T08:24:00.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:24:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0[47710]: debug 2024-01-21T08:24:00.048+0000 7fc09008e700 -1 osd.0 36 *** Got signal Terminated *** 2024-01-21T08:24:00.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:24:00 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0[47710]: debug 2024-01-21T08:24:00.048+0000 7fc09008e700 -1 osd.0 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-21T08:24:00.490 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jan 21 08:24:00 smithi046 podman[74401]: ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.0 2024-01-21T08:24:01.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.0.service' 2024-01-21T08:24:01.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:24:01.313 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-01-21T08:24:01.314 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-01-21T08:24:01.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.1 2024-01-21T08:24:01.592 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:24:01 smithi046 systemd[1]: Stopping Ceph osd.1 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:24:01.592 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:24:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1[50858]: debug 2024-01-21T08:24:01.447+0000 7fa9850cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-21T08:24:01.592 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:24:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1[50858]: debug 2024-01-21T08:24:01.447+0000 7fa9850cb700 -1 osd.1 36 *** Got signal Terminated *** 2024-01-21T08:24:01.593 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:24:01 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1[50858]: debug 2024-01-21T08:24:01.447+0000 7fa9850cb700 -1 osd.1 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-21T08:24:01.937 INFO:journalctl@ceph.osd.1.smithi046.stdout:Jan 21 08:24:01 smithi046 podman[74580]: ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.1 2024-01-21T08:24:02.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.1.service' 2024-01-21T08:24:02.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:24:02.681 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-01-21T08:24:02.681 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-01-21T08:24:02.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.2 2024-01-21T08:24:02.990 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:24:02 smithi046 systemd[1]: Stopping Ceph osd.2 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:24:02.990 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:24:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2[54012]: debug 2024-01-21T08:24:02.813+0000 7f96ebfce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-21T08:24:02.990 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:24:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2[54012]: debug 2024-01-21T08:24:02.813+0000 7f96ebfce700 -1 osd.2 36 *** Got signal Terminated *** 2024-01-21T08:24:02.990 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:24:02 smithi046 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2[54012]: debug 2024-01-21T08:24:02.813+0000 7f96ebfce700 -1 osd.2 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-21T08:24:03.337 INFO:journalctl@ceph.osd.2.smithi046.stdout:Jan 21 08:24:03 smithi046 podman[74761]: ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.2 2024-01-21T08:24:04.041 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.2.service' 2024-01-21T08:24:04.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:24:04.082 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-01-21T08:24:04.082 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-01-21T08:24:04.082 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.3 2024-01-21T08:24:04.389 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:24:04 smithi078 systemd[1]: Stopping Ceph osd.3 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:24:04.389 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:24:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3[60970]: debug 2024-01-21T08:24:04.194+0000 7fb78f009700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-21T08:24:04.389 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:24:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3[60970]: debug 2024-01-21T08:24:04.194+0000 7fb78f009700 -1 osd.3 36 *** Got signal Terminated *** 2024-01-21T08:24:04.389 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:24:04 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3[60970]: debug 2024-01-21T08:24:04.194+0000 7fb78f009700 -1 osd.3 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-21T08:24:04.741 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jan 21 08:24:04 smithi078 podman[69691]: ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.3 2024-01-21T08:24:05.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.3.service' 2024-01-21T08:24:05.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:24:05.470 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-01-21T08:24:05.470 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-01-21T08:24:05.470 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.4 2024-01-21T08:24:05.742 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:24:05 smithi078 systemd[1]: Stopping Ceph osd.4 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:24:05.742 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:24:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4[64226]: debug 2024-01-21T08:24:05.610+0000 7f3bc41ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-21T08:24:05.742 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:24:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4[64226]: debug 2024-01-21T08:24:05.610+0000 7f3bc41ba700 -1 osd.4 36 *** Got signal Terminated *** 2024-01-21T08:24:05.742 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:24:05 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4[64226]: debug 2024-01-21T08:24:05.610+0000 7f3bc41ba700 -1 osd.4 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-21T08:24:06.110 INFO:journalctl@ceph.osd.4.smithi078.stdout:Jan 21 08:24:05 smithi078 podman[69871]: ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.4 2024-01-21T08:24:06.813 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.4.service' 2024-01-21T08:24:06.852 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:24:06.852 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-01-21T08:24:06.852 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-01-21T08:24:06.852 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.5 2024-01-21T08:24:07.200 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:24:06 smithi078 systemd[1]: Stopping Ceph osd.5 for a17ce476-b833-11ee-95b0-87774f69a715... 2024-01-21T08:24:07.200 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:24:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5[67357]: debug 2024-01-21T08:24:06.994+0000 7f55d1600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-21T08:24:07.201 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:24:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5[67357]: debug 2024-01-21T08:24:06.994+0000 7f55d1600700 -1 osd.5 36 *** Got signal Terminated *** 2024-01-21T08:24:07.201 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:24:06 smithi078 ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5[67357]: debug 2024-01-21T08:24:06.994+0000 7f55d1600700 -1 osd.5 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-21T08:24:07.491 INFO:journalctl@ceph.osd.5.smithi078.stdout:Jan 21 08:24:07 smithi078 podman[70055]: ceph-a17ce476-b833-11ee-95b0-87774f69a715-osd.5 2024-01-21T08:24:08.285 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a17ce476-b833-11ee-95b0-87774f69a715@osd.5.service' 2024-01-21T08:24:08.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T08:24:08.324 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-01-21T08:24:08.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a17ce476-b833-11ee-95b0-87774f69a715 --force --keep-logs 2024-01-21T08:24:11.331 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a17ce476-b833-11ee-95b0-87774f69a715 --force --keep-logs 2024-01-21T08:24:33.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-21T08:24:33.439 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-21T08:24:33.469 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-21T08:24:33.471 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259/remote/ubuntu@smithi046.front.sepia.ceph.com/crash 2024-01-21T08:24:33.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/crash -- . 2024-01-21T08:24:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T08:24:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2024-01-21T08:24:33.524 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259/remote/ubuntu@smithi078.front.sepia.ceph.com/crash 2024-01-21T08:24:33.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/crash -- . 2024-01-21T08:24:33.571 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/a17ce476-b833-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T08:24:33.571 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2024-01-21T08:24:33.572 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-21T08:24:33.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-21T08:24:33.606 INFO:teuthology.orchestra.run.smithi046.stderr:grep: /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph.log: No such file or directory 2024-01-21T08:24:33.608 INFO:tasks.cephadm:Compressing logs... 2024-01-21T08:24:33.608 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T08:24:33.650 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T08:24:33.678 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T08:24:33.678 INFO:teuthology.orchestra.run.smithi078.stderr:: No such file or directory 2024-01-21T08:24:33.678 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T08:24:33.678 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mgr.smithi078.kcfjgo.log 2024-01-21T08:24:33.678 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.3.log 2024-01-21T08:24:33.678 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mgr.smithi078.kcfjgo.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mon.smithi078.log 2024-01-21T08:24:33.679 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.4.log 2024-01-21T08:24:33.679 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mon.smithi078.log: /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.5.log 2024-01-21T08:24:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T08:24:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:: No such file or directory 2024-01-21T08:24:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T08:24:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.2.log 2024-01-21T08:24:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mon.smithi046.log 2024-01-21T08:24:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.0.log 2024-01-21T08:24:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mon.smithi046.log: gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.1.log 2024-01-21T08:24:33.681 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.5.log: 89.6% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T08:24:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.0.log: /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.1.log: 89.8% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T08:24:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mgr.smithi046.lmxnwc.log 2024-01-21T08:24:33.683 INFO:teuthology.orchestra.run.smithi078.stderr: 92.6% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mgr.smithi078.kcfjgo.log.gz 2024-01-21T08:24:33.687 INFO:teuthology.orchestra.run.smithi078.stderr: 95.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T08:24:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mgr.smithi046.lmxnwc.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T08:24:33.713 INFO:teuthology.orchestra.run.smithi078.stderr: 93.3% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.5.log.gz 2024-01-21T08:24:33.727 INFO:teuthology.orchestra.run.smithi078.stderr: 93.2% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.4.log.gz 2024-01-21T08:24:33.741 INFO:teuthology.orchestra.run.smithi078.stderr: 92.5% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.3.log.gz 2024-01-21T08:24:33.753 INFO:teuthology.orchestra.run.smithi046.stderr: 93.0% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.2.log.gz 2024-01-21T08:24:33.758 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.0.log.gz 2024-01-21T08:24:33.767 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-osd.1.log.gz 2024-01-21T08:24:33.783 INFO:teuthology.orchestra.run.smithi078.stderr: 93.0% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mon.smithi078.log.gz 2024-01-21T08:24:33.784 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-01-21T08:24:33.784 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m0.117s 2024-01-21T08:24:33.784 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m0.259s 2024-01-21T08:24:33.785 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.030s 2024-01-21T08:24:33.856 INFO:teuthology.orchestra.run.smithi046.stderr: 93.1% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mgr.smithi046.lmxnwc.log.gz 2024-01-21T08:24:34.075 INFO:teuthology.orchestra.run.smithi046.stderr: 90.4% -- replaced with /var/log/ceph/a17ce476-b833-11ee-95b0-87774f69a715/ceph-mon.smithi046.log.gz 2024-01-21T08:24:34.076 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-21T08:24:34.076 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m0.409s 2024-01-21T08:24:34.076 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m0.791s 2024-01-21T08:24:34.077 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.055s 2024-01-21T08:24:34.077 INFO:tasks.cephadm:Archiving logs... 2024-01-21T08:24:34.077 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259/remote/ubuntu@smithi046.front.sepia.ceph.com/log 2024-01-21T08:24:34.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T08:24:34.258 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259/remote/ubuntu@smithi078.front.sepia.ceph.com/log 2024-01-21T08:24:34.259 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T08:24:34.333 INFO:tasks.cephadm:Removing cluster... 2024-01-21T08:24:34.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a17ce476-b833-11ee-95b0-87774f69a715 --force 2024-01-21T08:24:34.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a17ce476-b833-11ee-95b0-87774f69a715 --force 2024-01-21T08:24:34.949 INFO:tasks.cephadm:Removing cephadm ... 2024-01-21T08:24:34.950 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T08:24:34.968 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T08:24:34.986 INFO:tasks.cephadm:Teardown complete 2024-01-21T08:24:34.986 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-01-21T08:24:35.211 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bd9fdabce54f4516913c130b68b94a9e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-01-21T08:24:35.214 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-21T08:24:35.225 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T08:24:35.237 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T08:24:35.237 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T08:24:35.260 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T08:24:35.330 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T08:24:35.338 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T08:24:35.339 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T08:24:35.341 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T08:24:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2024-01-21T08:24:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:bash: ntpq: command not found 2024-01-21T08:24:35.362 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T08:24:35.362 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-01-21T08:24:35.362 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +167us[ +163us] +/- 81ms 2024-01-21T08:24:35.362 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -870us[ -874us] +/- 62ms 2024-01-21T08:24:35.362 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +918us[ +913us] +/- 76ms 2024-01-21T08:24:35.363 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:24:35.364 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T08:24:35.364 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-01-21T08:24:35.364 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +161us[ +161us] +/- 81ms 2024-01-21T08:24:35.364 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -877us[ -883us] +/- 62ms 2024-01-21T08:24:35.365 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +941us[ +941us] +/- 76ms 2024-01-21T08:24:35.365 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T08:24:35.365 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T08:24:35.374 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T08:24:35.374 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T08:24:35.388 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T08:24:35.472 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T08:24:35.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T08:24:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T08:24:35.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T08:24:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T08:24:35.596 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T08:24:35.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T08:24:35.620 INFO:teuthology.task.internal:Duration was 1794.606973 seconds 2024-01-21T08:24:35.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T08:24:35.635 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T08:24:35.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T08:24:35.637 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T08:24:35.683 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T08:24:35.684 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T08:24:36.023 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T08:24:36.023 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-01-21T08:24:36.023 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T08:24:36.048 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-01-21T08:24:36.048 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T08:24:36.072 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T08:24:36.072 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T08:24:36.090 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T08:24:36.163 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T08:24:36.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T08:24:36.166 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T08:24:36.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T08:24:36.279 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T08:24:36.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T08:24:36.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T08:24:36.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T08:24:36.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T08:24:36.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T08:24:36.374 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-01-21T08:24:36.400 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-01-21T08:24:36.420 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T08:24:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T08:24:36.464 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T08:24:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T08:24:36.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T08:24:36.491 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T08:24:36.492 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259/remote/smithi046 2024-01-21T08:24:36.493 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T08:24:36.546 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259/remote/smithi078 2024-01-21T08:24:36.547 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T08:24:36.585 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T08:24:36.585 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T08:24:36.588 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T08:24:36.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T08:24:36.648 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T08:24:36.648 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T08:24:36.660 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T08:24:36.661 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T08:24:36.663 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T08:24:36.678 INFO:teuthology.orchestra.run.smithi046.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 08:24 /home/ubuntu/cephtest 2024-01-21T08:24:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 08:24 /home/ubuntu/cephtest 2024-01-21T08:24:36.696 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T08:24:36.705 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T08:24:36.733 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T08:24:36.752 DEBUG:teuthology.nuke:shortname: smithi046 2024-01-21T08:24:36.753 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T08:24:36.777 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:45:56.367258', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErmfK9ULo+YAVrDB/iWuZ9mEXrI/tSfGQ79aCe8Jdwcwh8rShrRBJ/taeB2cBaNGhxolrBtlfEAU19Dp3gh3/Y='} 2024-01-21T08:24:36.780 DEBUG:teuthology.nuke:shortname: smithi078 2024-01-21T08:24:36.780 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T08:24:36.804 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:45:56.368679', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFmBer2yio2wXRcDtxhYMDLIm+dz3HdB2yr808l5E8NhhkCDQSgjH08RqWzHwrlWNoqYvSDUB6+Q18ZwcZPirE='} 2024-01-21T08:24:36.832 INFO:teuthology.orchestra.console.smithi046:Power off 2024-01-21T08:24:36.832 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T08:24:36.853 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-01-21T08:24:36.853 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T08:24:36.870 INFO:teuthology.orchestra.console.smithi078:Power off 2024-01-21T08:24:36.870 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T08:24:36.880 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-01-21T08:24:36.893 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-01-21T08:24:36.893 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T08:24:36.915 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-01-21T08:24:40.881 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T08:24:40.917 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T08:24:40.993 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-01-21T08:24:41.027 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-01-21T08:24:44.997 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T08:24:45.028 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T08:24:45.107 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-01-21T08:24:45.137 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-01-21T08:24:45.208 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-01-21T08:24:49.141 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T08:24:49.250 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-01-21T08:24:49.351 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-01-21T08:24:49.456 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1794.6069729328156 failure_reason: timeout expired in wait_until_healthy flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bd9fdabce54f4516913c130b68b94a9e status: fail success: false 2024-01-21T08:24:49.456 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T08:24:49.540 INFO:teuthology.run:FAIL